Counting on AllocVec's size memo..

[itix]

5. OS4 tries to enforce certain programming rules and punishes hacking.

Still many OS4 programs depend on undocumented side effects and features ;^) It is nature of programming. Coders always did and will use all means to reach goal with minimal effort. If it is possible to exploit an undocumented feature there is always someone doing so.

[Piru]

I don't say "let all the old applications crash and burn". I say "keep (wrong) compatibility as long as it is bareable, but sacrifice it if that would prevent really necessary improvements". And the new memory system really is a big step forward!

While not commenting the qualities of the new memory system, nothing in it required dropping the backwards compatibility here. And this is my point.

during all the beta testing of OS4 GoldED really was the *only* bad application.

It was? According to who? How broad set of applications were tested? How was it made sure that all memory allocations were tested in each app?

I think the Friedens would have decided the other way if more than this single program had been affected. But that was not the case.

Well, if they really think they caught the only program abusing this trick, they are gravely wrong.

[itix]

Ok, the location of AllocVec()'s size storage has become something like a "fact". But that's the point: it never really was a fact

This could be compared to how some OS4 programs handle r2 register. AmigaOS 4 uses SysV ABI and according to specification general purpose register r2 is reserved for system. However it is publicly known that r2 is not used by the system at all. Thus using r2 as a spare register in assembler routines is safe and I know OS4 native programs doing so. But this is not stated in development documents, "everyone" just know this feature.

Is it bad? I don't know. In MorphOS, killing r2 is illegal, but os4emu (;-)) occasionally abuses the system and kills r2 against all rules :-)

[weirdami]

Naturally applications relying on this are broken aswell, but there was no need to reduce the compatiblity in this way.

People made the same kind of complaint when AGA came out because lots of games stopped working.

[Karlos]

"Killing" any register not known to be defined as volatile on any specification is bad. Do what the hell you want to a register in your own code but be sure to save/restore anything not volatile.

Unless you are a bad coder of the type that makes dangerous assumptions about everything you come across.

I think there's a touch of melodrama in this thread. Some applications may break because they were badly written or had at least one majorly bad assumption.

Any code that makes these kind of assumptions will eventually fall over as the system moves on. There are applications that bail on every major operating system at some stage due to their age. Saying this is always the fault of the OS is a bit unfair when it is down to the actions of the developer.

Would you blame the C/C++ compiler's compatibility here or the developer if we substituted the word "AllocVec" with "malloc" or "operator new[]()" ?


If we look at the issues in context for a moment. There are apps that won't work. If they are very old, the chances are the services they provide have been replaced by newer applications. If they are new apps, the chances are they can  be fixed.

Similar problems with compatibility existed moving from 1.3 to 2.0 and that was a much smaller technical step than the changes from 3.9 to OS4 / MOS.

[itix]

"Killing" any register not known to be defined as volatile on any specification is bad. Do what the hell you want to a register in your own code but be sure to save/restore anything not volatile.

r2 is quite exceptional: it is either used or not. Killing r2 is not very expectional either. It is just about is it good coding style or not - while using r2 for own purposes is maybe deprecated its functionality is known (i.e. unused in OS4).

Would you blame the C/C++ compiler's compatibility here or the developer if we substituted the word "AllocVec" with "malloc" or "operator new[]()" ?

AllocVec() stayed the same last 15 years. Memory lists have existed last 20 years. I'd say API was rock solid many years...

Similar problems with compatibility existed moving from 1.3 to 2.0 and that was a much smaller technical step than the changes from 3.9 to OS4 / MOS.

In those times software was developed actively and companies could not let their software to fail under 2.0 especially because productivity users went for 2.0. Today I dont see software companies (what companies?) recompiling their software for MOS and OS4.

Gamers despised 2.0 and A500+ was considered being bad buy because of poor compatibility. A1200 users could not play many games until proper tools (WHDLoad and JST) were invented and games patched.

Of course something must be sacrificed always. I know why Diavolo Backup 68k doesn't work fully in MOS and OS4 and it is only because coder did extremely small mistake in threading code. It could be fixed by small adjust in the DOS side though.

[Karlos]

@itix

It appears you missed my point. Regardless of how long AllocVec() behaved the way it did and regarless of how long memory lists are stored it is *not* within the developers remit to assume anything about it.

A given compiler may handle malloc() the same way. If it did, for years and years, would you regard it as acceptable practise to operate on the assumption and use it within your code? I certainly would not, under any circumstances, ever.

If you need to care about the allocation size, perhaps in some system with a lot of dynamic allocation going on, then you wrap the allocator routines yourself or use pools, you sure as hell don't go around poking into negative pointer offsets from the base address, regardless of how long it might have worked historically. You sure as hell wouldn't do it in ANSI-C/C++, there's no reason to do it when using exec either. People only get tempted to do this because most of the structures used by the OS are visible in one header or another and fail to appreciate that they are private to the system regardless.

The point has been raised that there was no need to do this and break compatibility and that for compatibility sake it could have been left alone. Regardless of wether this is true or not, I think criticising the OS in this case is a bit of a strawman attack. The real issue is that developers shouldn't habitually point loaded guns at their own feet and wonder why they might lose some toes one day ;-)

[Piru]

So, OS4 "Final Update" with the new memory system is out.

I suppose no important software has suddenly broken with it?

[platon42]

Piru wrote:
So, OS4 "Final Update" with the new memory system is out.

I suppose no important software has suddenly broken with it?

Uhm, you mean like FinalWriter?

http://amigaworld.net/modules/newbb/viewtopic.php?mode=viewtopic&topic_id=21726&forum=32&viewmode=flat&order=0

[Stevo]

Come on, Final Writer '97 isn't that important: I'm sure AmigaWriter2 still works ;-)

[Karlos]

I have to say, this is pretty bad :-(

I still stand by the view that the developer was at fault for assuming things about the underlying memory subystem. There's no need for it. Certainly I've never needed to. Feel free to prove me wrong on that count. For a major application, this practise is inexcusable.

Amiga developers need to understand what APIs are and when they must use them. Banging the metal or underlying OS implementation internals may have been acceptable in the 80's but things change.

[Piru]

@Karlos

Of course the developer of the app is at fault here, too.

But so are OS4 developers. In insisting on removing the compatibility (even if undocumented!) they've broken important application that worked just fine before. Again, there is no good reason for not to have the compatibility there. Removing it will only break apps.

Reading the local forum, SnoopDOS, AmiDisk, HDRec are all crashing.

QED

[whose]

@piru:

>Reading the local forum, SnoopDOS, AmiDisk, HDRec are all
>crashing.

Neither of them do. SnoopDos68K runs in fact "better" than SnoopDosOS4 (you can't open the "Functions" window anymore).

That results from a nice "trick", doing some write accesses to const data sections. Nice, eh? SnoopDos68K doesn't break because 68K programs are still allowed to do this.

AmiDisk works fine, HDRec does so, too (in fact, now you can use certain functions to the extent, because Petunia runs even faster with Final, about 2 times faster).

What breaks now is FinalWriter, WordWorth7 is doing its Job fine here. Even TurboPrint works flawlessy here (with reset proof option disabled).

If one looks at the practice to rely on the memory size notation of AllocMem(), one must say: Why do you do this? It's "forbidden", there is no real problem to do own notation of the alloc'd size, just do it the "right" way.

There is no need to stay compatible here. If older applications break for this, update the application or make an alternative for it.

Greetz

[Piru]

Regarding crashing applications: Those apps do crash for the local OS4 user. I have no way to verify those findings.

If one looks at the practice to rely on the memory size notation of AllocMem(), one must say: Why do you do this? It's "forbidden", there is no real problem to do own notation of the alloc'd size, just do it the "right" way.

It was about AllocVec, but still: I agree fully. Programmers should use the correct way and avoid depending on the size being at -4. But they don't. There are such apps, and not all of them can be replaced.

There is no need to stay compatible here.

There is no reason to remove compatibility here.

If older applications break for this, update the application or make an alternative for it.

And if the source code is not available and the application is some important one? You seriously think everything can be replaced? AROS / OS4 / MOS doesn't have enough good developers to write replacement for everything.

[whose]

@piru:

It was about AllocVec, but still: I agree fully.

Yes, my fault. Was in a hurry

Programmers should use the correct way and avoid depending on the size being at -4. But they don't. There are such apps, and not all of them can be replaced.

Well, they "didn't". Nowadays developers should avoid even to ask for this. Well, they don't, I know. But where's the point to discuss the compatibility question for older programs, when a programmer actually asks for this OS private special? Newer applications should avoid such tricks for any price! There is no excuse to rely on such a "feature" nowadays.

There is no reason to remove compatibility here.

I bet there is. But I'm not "skilled" enough to judge or explain it. The Frieden brothers can explain it much better, I think.

I don't rely on the size notation of AllocVec() and I don't even ask for it. I simply don't need it. But there are enough "tricks" to achieve the goal in another, OS friendly way (the mentioned wrapper, just notice the size at the beginning of the wanted memory space + (type)1 and give back the address + (type)1. This is the simplest way, there are more clever ways to do this).

And if the source code is not available and the application is some important one? You seriously think everything can be replaced? AROS / OS4 / MOS doesn't have enough good developers to write replacement for everything.

Well, you don't have to replace everything. There are very few apps that break. Even for SnoopDos there is a replacement now. So, you don't need so much developers to write replacements.

But the will to do so is quite small in most cases. Mostly developers find it easier to e.g. ask for AllocVec() specials to invest lots of work to get such stunts compatible for all known systems. Not very clever, I think.

Regarding crashing applications: Those apps do crash for the local OS4 user. I have no way to verify those findings.

Yes, I know this. If it would be possible I would invite you here to Germany to see it working yourself.

Some other users had problems which I helped to solve (especially on AW). Much of their problems resulted from not obeying the installation instructions (clean install) or failure of 3rd party programs (especially IBrowse installing old MUI classes, but some older libraries made problems, too, which could have been avoided by using the newest versions, for example expat.library).

For HDRec you could ask the author, he obviously did some speed tests for HDRec with OS4 final and gives the same results I achieved.

[itix]

Neither of them do. SnoopDos68K runs in fact "better" than SnoopDosOS4 (you can't open the "Functions" window anymore).

That results from a nice "trick", doing some write accesses to const data sections. Nice, eh? SnoopDos68K doesn't break because 68K programs are still allowed to do this.

Program overwriting its own const data is not illegal as such and in SnoopDos it happens only because some strings are unexpectedly placed into .rodata section. There is nothing wrong in SnoopDos 68k; bug was created in 68k->PPC transition. Only PPC version of SnoopDos (inluding MorphOS port) writes to const data.

Fixing const data problem does not give you working packet monitor though.

If one looks at the practice to rely on the memory size notation of AllocMem(), one must say: Why do you do this? It's "forbidden", there is no real problem to do own notation of the alloc'd size, just do it the "right" way.

Even OS4 developers do it. DiskSpeed OS4 assumes that memory returned from AllocVec() is aligned to 8 bytes boundary. It is not documented feature but is "public knowledge". Neither is documented what newlib startup code (one that is statically linked to newlib programs) is doing. Build hello world program and link it against newlib and you are already using undocumented features.