The OS4 direction thread

[Matt_H]

As some people may have read my comments in the recent news announcement, I'm a little worried that OS4 is taking a sharp turn towards Unix. In the interests of not getting that announcement too cluttered up with a side discussion, I'm continuing said discussion here.

After three years in demi-release, I think it's time for some more dedicated software to appear for OS4. Quick and dirty ports are fine as a stopgap measure to get much-needed core programs available, but building the entire OS4 software catalog out of them is a very dangerous move. Why should I run OS4 when I can run Linux if there's nothing unique about the programs it runs?

"ZOMG st0p teh ports!!11 we need teh rael am1ga softwears!!11" - No, that's not what I'm saying at all. Ports are great if they're well and truly ported - copy the ScummVM options pages into pulldown menus, integrate a GUI into shell tools, change the commandline arguments from -options to OPTIONS/S. Improvements like these give the Amiga versions an edge and make them a greater asset for the platform.

See the MorphOS version of SDL for a fantastic example of how things could be done: shared libraries, and system-wide and per-application preferences. In terms of native OS4 software, Wet is another great example of Amiga programming as it takes advantage of dockies, AppIcons, and ENV vars.

Bottom line, I don't want to run Firefox on my Amiga, I want to run the Amiga version of Firefox. If the shared objects help get Firefox running initially, that's great, but my concern is that the shared objects wil discourage developers from actually Amiga-izing their ports and that such a thing will turn us into a "me too" platform.

[Hans_]

@Matt_H

Rogue made it clear (somewhere) that shared objects are not going to replace shared libraries. Shared objects have no way to share data between multiple processes; shared libraries do.

In fact, it's this difference that makes porting shared objects to libraries a pain. Shared objects can use local variables without screwing up multiple processes because local variables aren't shared across multiple processes (the MMU is used to create local versions, I've been told). With shared libraries, such variables are shared across all apps using that library, making subtle bugs occur. This means that maintaining a port takes a lot of effort. Every new version must be checked/repatched for these kinds of issues.

I don't think that it's turning Amiga OS into another Unix system. You can't use Linux PPC shared objects in Amiga OS. Yes, it does mean that we're going to get more quick and dirty ports. One thing you can rely on though, people will probably still work on OS4 native software simply because they don't like the quick and dirty ports.

Hans
 

[Piru]

It's rather weird that OS4 coders are incapable of creating library with per-task sdata. It's rather trivial with -mresident32 and small r13-restoring code snippet. With such code, only the library base itself and static variables need to be allocated per library opener. This is the most efficient and natural way to handle it, and it's far from painful.

Funniest comments regarding this have come from some guys claiming various MorphOS programs are statically linked due to missing "feature", when in fact MorphOS has had it for years, and implemented properly.

[Hans_]

@Piru

Should people really be doing this in shared libraries/objects?

int func()
{
    static int staticLocal;

    // further code that uses staticLocal
}

Apparently that's quite ok with a shared object, because each process that links to the shared object has it's own local var space. To me, it just looks like a thread unsafe time-bomb waiting to happen. It's bad coding practise. What happens if you use such code from multiple threads within the same process?

I like the fact that I can decide what is global and what is  local to a library opener. My understanding is that this is why they created the Open/CloseInterface() system in OS4 (you can have per interface data without messing with the lib-base).

Really, I think that they created shared-object loading facilities so that you can just take some cross-platform source code, type make, and build the app, without having to write library frameworks (a lot of extra work). It's also less of a problem if someone adds/removes functions to/from the shared-object (I think that removing functions from a library after release is a bad idea, you're breaking compatibility). Then there's the issue of some shared-objects wanting to link to external global variables. I can't think of any clean way of doing that with a library (I also don't think that a library should have external dependencies like that).

Hans

[Piru]

@Hans_

Apparently that's quite ok with a shared object, because each process that links to the shared object has it's own local var space. To me, it just looks like a thread unsafe time-bomb waiting to happen. It's bad coding practise. What happens if you use such code from multiple threads within the same process?

Depends. If you're on unixy system, it works out of the box. On MorphOS you can adjust the code or use relskeleton library model.

In many cases you're not writing the whole library from scratch. Quite a lot of lib code assumes that it is not shared between processes, that there is a separate memory space.

If you're writing your own code, you obviously can do the thing properly from the beginning, but often this is not the case. You can use the MorphOS relskeleton library model easier than actually rewriting tons of unfamiliar code.

I like the fact that I can decide what is global and what is local to a library opener.

This can be done easily with the relskeleton model. By default all static variables are local. To get "real" global variable just place it to section other than .sdata:

Code: [Select]

int reallyglobal __attribute__ ((section (".realsdata")));
Also, another neat feature of the thing is that you can have a "init-once" variables, that is that any local variable can be preset at LibInit, and it will be copied to all children-libraries. This is quite nice for library bases and such.

Really, I think that they created shared-object loading facilities so that you can just take some cross-platform source code, type make, and build the app

Well, you don't really get very well functioning stuff that way. And what happened to ixemul anyway? It actually works amazingly well on many such occasions (and you hardly ever really absolutely need shared-object stuff in quick compiles anyway).

without having to write library frameworks (a lot of extra work).

Actually the relskeleton framework is quite simple. Also, it doesn't involve changing millions places in the sourcecode (However, naturally some sources require more work than others). I'll much rather take proper library that doesn't gobble tons of memory and resources just because coders were lazy and/or incompetent.

It's also less of a problem if someone adds/removes functions to/from the shared-object (I think that removing functions from a library after release is a bad idea, you're breaking compatibility).

Huh. You never remove functions or insert new ones before the old. You only declare them obsolete, and wrap the old API so that it can use the services of the new, thus keeping binary compatibility. If the API changes completely, just create a new library (change the name) instead. This should be quite obvious.

Then there's the issue of some shared-objects wanting to link to external global variables. I can't think of any clean way of doing that with a library (I also don't think that a library should have external dependencies like that).

I'm not quite sure what you mean by that, but assuming it's just having some "really" global variable: the section trick does the job just fine, if you really must. If you just need some variable from other loadable component, you can do this quite easily too. There are various way to do this, for example fields in public library possize, or some function to return the variable pointer.

And finally: This relskeleton stuff is nothing magic, it's actually old news. Even SAS/C did it decades ago already.

I frankly don't understand why OS4 can't have the same.

[Hans_]

Piru wrote:
@Hans_

Really, I think that they created shared-object loading facilities so that you can just take some cross-platform source code, type make, and build the app

Well, you don't really get very well functioning stuff that way. And what happened to ixemul anyway? It actually works amazingly well on many such occasions (and you hardly ever really absolutely need shared-object stuff in quick compiles anyway).

Ixemul doesn't provide you with all the libs that a Unix system has. I seem to recall there being issues with ixemul on OS4, but I don't know whether that's been fixed or not.

BTW, why won't you get well functioning stuff this way? It works on Linux. The current X11 Firefox compile for Amiga OS4 is about 118 MB thanks to everything being statically linked (and compiled with the debug flag). X11 + apps is also sizeable because it's all statically linked. Having them use shared objects will at least not waste my HD space.

without having to write library frameworks (a lot of extra work).

Actually the relskeleton framework is quite simple. Also, it doesn't involve changing millions places in the sourcecode (However, naturally some sources require more work than others). I'll much rather take proper library that doesn't gobble tons of memory and resources just because coders were lazy and/or incompetent.

I know it's not complicated to write an Amiga library wrapper framework, but it takes a lot longer than simply typing make and hitting the enter key.

It's also less of a problem if someone adds/removes functions to/from the shared-object (I think that removing functions from a library after release is a bad idea, you're breaking compatibility).

Huh. You never remove functions or insert new ones before the old. You only declare them obsolete, and wrap the old API so that it can use the services of the new, thus keeping binary compatibility. If the API changes completely, just create a new library (change the name) instead. This should be quite obvious.

Maybe YOU don't. However, that's the reason why sometimes v5 of a particular .so is incompatible to v4. I think it's a really stupid thing to do as this means you need new versions of all apps that used that .so (not a big problem if it's all open source, but it's still not good). Some people do mess with functions in shared objects this way.

Then there's the issue of some shared-objects wanting to link to external global variables. I can't think of any clean way of doing that with a library (I also don't think that a library should have external dependencies like that).

I'm not quite sure what you mean by that, but assuming it's just having some "really" global variable: the section trick does the job just fine, if you really must. If you just need some variable from other loadable component, you can do this quite easily too. There are various way to do this, for example fields in public library possize, or some function to return the variable pointer.

What I mean is that shared objects can also have, for example, something like: extern int appGlobal. The application would be expected to provide this global variable.

And finally: This relskeleton stuff is nothing magic, it's actually old news. Even SAS/C did it decades ago already.

I frankly don't understand why OS4 can't have the same.

There's no technical reason why they couldn't. Personally, I prefer shared libraries to shared-objects, but I don't have a problem with them providing shared-objects for those porting software over. If it takes them less time porting stuff over, maybe they can spend more time writing real Amiga software.

Hans

[Piru]

@Hans_

Ixemul doesn't provide you with all the libs that a Unix system has.

And it never was supposed to. You're supposed to download the src, unpack, configure and make. Yes, static libraries mostly, but for quick'n'dirty stuff it works just fine. If you use the particular library in many things, you can make it real shared library easily.

Having them use shared objects will at least not waste my HD space.

But it's quite trivial to have both small disk and memory usage. So why not use the best method?

I know it's not complicated to write an Amiga library wrapper framework, but it takes a lot longer than simply typing make and hitting the enter key.

It depends on what you want.

If you want amiga application, you need to work on it anyway, just doing simple configure + make doesn't give you a working Firefox.

Anyway, relskeleton example library is available from the MorphOS Developer Connection. It's simple copypaste job to adjust it for whatever project needed. It really is rather easy stuff, and the benefits IMO easily make it worth the effort.

Huh. You never remove functions or insert new ones before the old. You only declare them obsolete, and wrap the old API so that it can use the services of the new, thus keeping binary compatibility. If the API changes completely, just create a new library (change the name) instead. This should be quite obvious.

Maybe YOU don't. However, that's the reason why sometimes v5 of a particular .so is incompatible to v4. I think it's a really stupid thing to do as this means you need new versions of all apps that used that .so (not a big problem if it's all open source, but it's still not good). Some people do mess with functions in shared objects this way.

relskeleton can actually help: The shared library can easily "drop" old incompatible library entries, and mark them obsolete, and then assign new function slot for the new call with the new API. Thus old applications will call the LVO of the old function, new applications will call the LVO of the new function. Both will continue to work. This works because the functions are not searched by name, but the function references are translated to offsets which will stay fixed. Obviously with a library API in constant flux this will explode the library-base quite fast, so at some point new library name migth be a good idea. So here shared library actually is beneficial.

What I mean is that shared objects can also have, for example, something like: extern int appGlobal. The application would be expected to provide this global variable.

Right. Such thing is pretty fubared up indeed, but luckily not that common in sensible APIs, most of such things are passed to the function in question as argument.

There's no technical reason why they couldn't.

Really? So why don't they? For example python would work perfectly fine as a real shared library, and then it would not eat as much memory, either.

MorphOS python binary is 9KB, while python.library is 1.2MB. The library is loaded to memory once.

[Hans_]

Piru wrote:
@Hans_

There's no technical reason why they couldn't.

Really? So why don't they? For example python would work perfectly fine as a real shared library, and then it would not eat as much memory, either.

MorphOS python binary is 9KB, while python.library is 1.2MB. The library is loaded to memory once.

Limited resources is the only reason I can think of. They have a limited team working on Amiga OS4, and a huge number of tasks. It's a tool for anyone porting stuff over; it saves time.

Hans