Amiga.org
Amiga computer related discussion => Amiga Software Issues and Discussion => Topic started by: Matt_H on October 31, 2010, 11:55:36 PM
-
I've just finished solving the OS4 equivalent of an episode of DLL hell. An issue in either DOpus 4.18.22 or elf.library 53.12 attempts to open libbz2.so.1.0, which doesn't exist on a 4.1u2 installation with the latest SDK (there's libbz2.so.1.0.4 instead), resulting in 2 highly uninformative error requesters. I was finally able to track down the culprits with the help of a Snoopy log file. xadmaster.library might be involved as well.
This is what happens when version strings are embedded in filenames and then hardcoded into programs. Too many files to keep track of, and they're easily lost during minor upgrades.
Has anyone else experienced this? Or, has anyone ever experienced anything like this with the traditional .library system?
-
It's up to the app to make meaningful error messages if LoadLibrary returns NULL. If it just exits with no error message, you're back to using snoopy to find which library failed.
-
You didn't even run into the fun scenario: Two incompatible SObjs with the exact same name, but different ABI (this has happened at least once already). There hasn't been proper documentation about SObj creation and this has lead into all sort of trouble.
They're worse than amiga shared libraries anyway: SObjs aren't even shared, each opener gets their own copy in memory.
As far as I can tell the only reason they exist is that apparently OS4 coders are incapable of creating amiga shared libraries with per caller library base (and the static data in the library base possize). Or they're just too lazy to do it properly, and decided to take the shortcut by introducing the non-shared libraries.
A grave mistake if you ask me.
-
incorporating version numbers into lib names seems to be a bad habit taken over from the systems incapable to provide version information ín another way.
-
As far as I can tell the only reason they exist is that apparently OS4 coders are incapable of creating amiga shared libraries with per caller library base (and the static data in the library base possize). Or they're just too lazy to do it properly, and decided to take the shortcut by introducing the non-shared libraries.
I think the rationale was for easy quick-n-dirty porting (which I'm also not fond of), but I'm thoroughly baffled as to why these well-established native programs are affected. OS4 xadmaster dates from December 2006, which I thought was before SObjs: was introduced. And I can't see why DOpus would need libbz2.so on its own.
incorporating version numbers into lib names seems to be a bad habit taken over from the systems incapable to provide version information ín another way.
Absolutely agreed. That's not a practice we should be emulating.
Compiling these ported libraries into programs statically would be a cleaner solution.
-
To my mind, the biggest problem with shared objects is that they suffer from NIH. However, since we all decided ELF was a good idea, it's not entirely unreasonable for someone to consider the linkable part of the specification.
You didn't even run into the fun scenario: Two incompatible SObjs with the exact same name, but different ABI (this has happened at least once already). There hasn't been proper documentation about SObj creation and this has lead into all sort of trouble.
Yeah, it was a pain back when most people were migrating from gcc 2.95, the ABI changed a great deal. However, one of the reasons it changed so much was to avoid having to do it again for a long, long time. Don't expect to see changes of that magnitude any time soon. Unless C++0x throws a big spanner in the works. Not impossible...
They're worse than amiga shared libraries anyway: SObjs aren't even shared, each opener gets their own copy in memory.
Sure, on AmigaOS, but not for OS that support .so files generally.
As far as I can tell the only reason they exist is that apparently OS4 coders are incapable of creating amiga shared libraries with per caller library base (and the static data in the library base possize). Or they're just too lazy to do it properly, and decided to take the shortcut by introducing the non-shared libraries.
The only reason? I think that's being deliberately short sighted. My guess would be that one really obvious reason would the fact that traditional amigaos shared libraries offer no way to provide the required linker constructs for C++ and other languages that some people, myself included, like to use. The world is not tied to procedural programming.
If someone wrote an improved .library specification and loader that allowed things like C++ symbol export it would surely have problems too and at least we know what .so's particular problems are.
A grave mistake if you ask me.
The only alternative that provides the same support is static linking. Assuming we disregard the increase in disk storage (which is cheap), one disadvantage of static linking is that you don't get any benefits of a new version of a library without re-linking your app. Obviously, any major change to said library, by definition, probably requires a recompile of the application, so no problem there. On the flipside, provided the exported symbol table doesn't change, there shouldn't need to be any such change. Theoretically, I should be able to swap out some generic .so with say an altivec optimised one, for example and all client users of that library get the same benefit.
-
I think the rationale was for easy quick-n-dirty porting (which I'm also not fond of), but I'm thoroughly baffled as to why these well-established native programs are affected. OS4 xadmaster dates from December 2006, which I thought was before SObjs: was introduced. And I can't see why DOpus would need libbz2.so on its own.
Agreed, that is pretty weird. A .so having a dependency on a .library is fine, the converse doesn't really seem reasonable when viewed on the "nativeness" scale in which .library is obviously further up the chain than .so.
Compiling these ported libraries into programs statically would be a cleaner solution.
Personally, I dislike static linking, but that's just me. It does make it easier for the end user, but then they always complain about the bloat of statically linked executables and the fact that statically linked application X doesn't support feature X that exists in application Y which happens to have been linked against a newer library :)
-
Yeah, it was a pain back when most people were migrating from gcc 2.95, the ABI changed a great deal. However, one of the reasons it changed so much was to avoid having to do it again for a long, long time. Don't expect to see changes of that magnitude any time soon. Unless C++0x throws a big spanner in the works. Not impossible...
Well actually what I was trying to say that there were two different libraries with different parameters, but the same name. Clearly this didn't end up well.
Sure, on AmigaOS, but not for OS that support .so files generally.
Obvously I was only talking about the shoddy SObjs implementation of AmigaOS here.
The only reason? I think that's being deliberately short sighted. My guess would be that one really obvious reason would the fact that traditional amigaos shared libraries offer no way to provide the required linker constructs for C++ and other languages that some people, myself included, like to use. The world is not tied to procedural programming.
C++ must die anyway ;)
-
Personally, I dislike static linking, but that's just me. It does make it easier for the end user, but then they always complain about the bloat of statically linked executables and the fact that statically linked application X doesn't support feature X that exists in application Y which happens to have been linked against a newer library :)
Yeah, in the past I have complained about the bloat of static linking, but as I think about it now, maybe it deserves more consideration. Ease of use for the end user used to be of paramount importance on the Amiga. And a static program is a product of the tools available at the time it was compiled. If the tools improve, recompiling the program to take active advantage of those tools might encourage continued developer involvement.
-
C++ must die anyway ;)
Get away with ye' , y' love it really :-)
Well, I do, and that's all that matters from this direction :D
-
And a static program is a product of the tools available at the time it was compiled.
This thus also means that if a security flaw in one of the libs that are linked to a lot of programs, they need to be relinked and redistributed to the user.
I know, mentioning security and Amiga in the same context does currently not make much sense.
greets,
Staf.
-
Personally, I'd love to see some native implementation supported by a decent compiler that would allow traditional shared libraries to provide the required services that .so files provide as far as languages beyond C are concerned.
However, since nobody has done anything like that (and it would not be a trivial undertaking), supporting the existing ELF shared object at least grants the developer some flexibility. Furthermore, it's not as alien a concept as it seems, seeing as we're using ELF already.
The earlier observation that a pair of identically named .so files implementing different ABIs being a uniquely bad problem seems a little contrived. After all, the existing .library mechanism doesn't prevent this kind of lunacy either. I could create a foo.library that is nothing to do with the existing foo.library and as long as the meta data is acceptable, the OS will happily pass it to clients that wanted the original, only to invoke meaningless vectors and crash.
-
This thus also means that if a security flaw in one of the libs that are linked to a lot of programs, they need to be relinked and redistributed to the user.
I know, mentioning security and Amiga in the same context does currently not make much sense.
greets,
Staf.
True, there is that risk. But I'll argue that we have a) security through obscurity, and b) that TCP/IP through a single system library offers an additional inherent layer of security. For example, look how much trouble we have getting Samba daemons and other servers working properly on home networks, let alone across the web :)
And if a developer needs to recompile to fix a security hole, maybe they'll also take the time to squash bugs or add new features.
The earlier observation that a pair of identically named .so files implementing different ABIs being a uniquely bad problem seems a little contrived. After all, the existing .library mechanism doesn't prevent this kind of lunacy either. I could create a foo.library that is nothing to do with the existing foo.library and as long as the meta data is acceptable, the OS will happily pass it to clients that wanted the original, only to invoke meaningless vectors and crash.
Also a possibility, but with particularly well-coded programs you can put necessary .library files in PROGDIR: or PROGDIR:Libs instead of SYS:Libs. Granted, that doesn't make it "shared" anymore, but you can have as many different incarnations of foo.library as you have programs that use it. I think MorphOS checks all these places (and more) for every library it attempts to open - Piru can correct me here. This sort of arrangement for .so files would be an improvement.
-
Also a possibility, but with particularly well-coded programs you can put necessary .library files in PROGDIR: or PROGDIR:Libs instead of SYS:Libs. Granted, that doesn't make it "shared" anymore, but you can have as many different incarnations of foo.library as you have programs that use it.
Mhh... well... if you don't do that multitasking thing, sure. And maybe remember to expunge the library once your program is done with it.
-
Also a possibility, but with particularly well-coded programs you can put necessary .library files in PROGDIR: or PROGDIR:Libs instead of SYS:Libs.
Sobjs have the same mechanism, IIRC.
-
Sobjs have the same mechanism, IIRC.
A mechanism that works better for sobjs than for shared libraries.
-
A mechanism that works better for sobjs than for shared libraries.
Of course, since you can have two programs using incompatible sobjs at the same time in memory and working.
Also, the "non-sharedness" of sobjs doesn't make them entirely useless, they save hard disk space even if they don't save memory (less useful, i admit), and they also still enable you to install a bugfixed library and not have to recompile all the apps that use it to gain the benefit.
Edit: oh.. and the amiga library system isn't immune to some of these problems, I keep two slightly incompatible versions of bgui.library handy, since I have two apps that work with one but not the other, and I can't run both at the same time.
-
Also a possibility, but with particularly well-coded programs you can put necessary .library files in PROGDIR: or PROGDIR:Libs instead of SYS:Libs. Granted, that doesn't make it "shared" anymore, but you can have as many different incarnations of foo.library as you have programs that use it. I think MorphOS checks all these places (and more) for every library it attempts to open - Piru can correct me here. This sort of arrangement for .so files would be an improvement.
That doesn't work. If you load application X which requires progdir:foo.library A and then start application Y which requires libs:foo.library B, application Y isn't going to work. The reason being that the request to open foo.library is going to note that it's already open and return the base pointer to it.
Contrast this to progdir:foo.so versus SObjs:foo.so. If the .so files are searched for in the same PROGDIR: > SOBJS: order, then both applications should run fine, each with it's own private copy of the library.
-
That doesn't work. If you load application X which requires progdir:foo.library A and then start application Y which requires libs:foo.library B, application Y isn't going to work. The reason being that the request to open foo.library is going to note that it's already open and return the base pointer to it.
Contrast this to progdir:foo.so versus SObjs:foo.so. If the .so files are searched for in the same PROGDIR: > SOBJS: order, then both applications should run fine, each with it's own private copy of the library.
As the author of an OS3.0 "particularly well-coded program" :D I am free to include any required library in PRODIR:libs/ under a different name such as ChaosFoo.library. Then everything is guaranteed to work.
I don't think I am currently doing this but I was forced to do it many years ago because sometimes a library evolves over time and becomes incompatible with different versions of itself.
-
That doesn't work. If you load application X which requires progdir:foo.library A and then start application Y which requires libs:foo.library B, application Y isn't going to work. The reason being that the request to open foo.library is going to note that it's already open and return the base pointer to it.
Contrast this to progdir:foo.so versus SObjs:foo.so. If the .so files are searched for in the same PROGDIR: > SOBJS: order, then both applications should run fine, each with it's own private copy of the library.
Interesting point. I hadn't thought about loading both libraries into memory.
So it looks like keeping shared objects in PROGDIR: instead is a realistic compromise between taking advantage of updated/bugfixed .so files without recompiling, and avoiding version control issues in SObjs:. Is that a fair assessment?
Some issues still outstanding are whether programs can take advantage of updated shared objects if the program in question ties them to a specific version by filename (and if it's possible to not do that), and what to do if PROGDIR: is C: (i.e., for ported command line utilities). Are the latter type of programs stronger candidates for static linking?
-
Edit: oh.. and the amiga library system isn't immune to some of these problems, I keep two slightly incompatible versions of bgui.library handy, since I have two apps that work with one but not the other, and I can't run both at the same time.
Yep, that's true. This is why I have a manually patched "agui.library" (and the corresponding application) and the newer bgui.library here. BTW: I you scan through your SOBJS: you might find some .so's that actually have a proper Amiga version string, which makes it easier for us as users. So it CAN be done - but is not usual at the moment. I guess this is one of the shortcomings that we do not have a proper and updated RKRM/style guide and coding rules.
-
I know it's sort of off topic, but have you guys played with Objective-C and its runtime object oriented system? I think it's wonderful to have your classes just like the old Amiga shared library :)
-
I know it's sort of off topic, but have you guys played with Objective-C and its runtime object oriented system? I think it's wonderful to have your classes just like the old Amiga shared library :)
It's the only inherent feature of Objective-C that I actually like. The rest of it is a dog's dinner :D
-
It's the only inherent feature of Objective-C that I actually like. The rest of it is a dog's dinner :D
Well, since I now have to use C++ for my microcontroller hobby, I have come tothe conclusion that, for high level App development I much prefer Obj-C... But low-level stuff is without question much better done in C++, objective C really wouldn't make much sense there at all!
-
Some issues still outstanding are whether programs can take advantage of updated shared objects if the program in question ties them to a specific version by filename (and if it's possible to not do that), and what to do if PROGDIR: is C: (i.e., for ported command line utilities). Are the latter type of programs stronger candidates for static linking?
You could static link against a library if you only wanted that particular version and no other.
Linux uses a versioned shared object scheme that works, but it does require programmers to observe a few rules. The actual .so file will contain a major and minor version number as part of it's name. Changes to the minor version number are not allowed to change the interface presented by the .so (well, they may be able to add new calls and data but they can't alter or remove any existing ones) , they are for internal changes only (bugfixes etc). Changes to the major version number indicate larger changes to the interface that may not be backwards compatible.
Normally, an application will be linked against a major version. Generally, this is just a symlink to a specific revision to that version. Updates can then come along and add new revisions. If any thing goes wrong with a new revision, rolling back to the previous one is trivial, the symlink is just pointed back at the last good working revision.
If you get a new major version, you still can have your previous one installed and all applications linked against that previous major version will still use it.
Finally, you usually have a simple .so symlink to the most recent major version/revision for applications which aren't supposed to be version sensitive.
It all sounds far more complex than it is, but it does mean you can create schemes with multiple versions/revisions of the same shared library installed that generally satisfies most applications. It used to be a bit of a pain but once a few guidelines were set it all sorted itself out.
I think the problem we have in OS4 right now is more a lack of best-practise when it comes to the creation and distribution of .so files.
-
Well, since I now have to use C++ for my microcontroller hobby, I have come tothe conclusion that, for high level App development I much prefer Obj-C... But low-level stuff is without question much better done in C++, objective C really wouldn't make much sense there at all!
Give it time. You will eventually realise, as I did, that it's all, without question, much better done in C++ ;)
Seriously though, you're right. C++ was always intended to be a superset of C and not intended to in any way reduce the language's suitability for system level programming, rather it was intended to make it more suitable for large scale application development.
-
That doesn't work. If you load application X which requires progdir:foo.library A and then start application Y which requires libs:foo.library B, application Y isn't going to work. The reason being that the request to open foo.library is going to note that it's already open and return the base pointer to it.
This is not something inherently connected to .library style libraries but to how the loading/caching is currently implemented in exec.
greets,
Staf.
-
This is not something inherently connected to .library style libraries but to how the loading/caching is currently implemented in exec.
That's a given, but it is implemented the way it is implemented. You can't have two slightly different versions of the same .library open concurrently, regardless of where you put them on your HD.