New improved intuition.library version from the Kickstart 3.1

[som99]

Upload in few minutes !

 intuition.library v40.86 beta 7 68020+ (101 024 bytes)

Thanks, downloaded and will patch later today

[warpdesign]

You see it too much as a engineer but AmigaOS on 68k is completely dead, there will be no development anymore there, noone paying for coordination. AmigaInc. are not interested in it, the same for Hyperion and the old sources sticking legally between both. That developers like you are hindered to help by old contracts just because they have had access to the old sources just show me that this community only has future if we move on with open sources (and there is only Aros) and forget all that were part of the past. For me AmigaOS 68k is dead and future only with Aros 68k. I know that some still think different but I hope people in the 68k community start to change mind in future.

I agree open source is the way. What I don't get is why other Amiga "flavors" do not embrace open source as well. They seem to be so socret & protective, like there were millions to be done with OS4/MOS...

[matthey]

It is optimal it is inlined. Actually, any decent compiler will that do automatically for you if the size of the memory block to be copied is known at compile time, including a complete unrolling of the copy if the size is small enough, and probably fall back to a small inlined routine if it is not avoiding the per-call overhead then. At least this is what SAS/C does, and it's quite plausible that this is the best option.  

I'm not so sure that "most" compilers will optimize memcpy() for static cases. Most compilers will optimize to some extent small copies implied with '=' which are practically all static sizes. The C memcpy() function is generally optimized for small to medium sized copies and doesn't detect for small static sizes. I have never seen the SAS/C memcpy()  function inlined (and it is poor for the 68040 and awful for the 68060). Vbcc will use assembler inlines by default for memcpy() and the new unreleased version of vbcc has much improved 68000 and 68020 optimized assembler versions of memcpy(). Maybe GCC is smart enough to check for static cases when using memcpy(). Medium to large sized copies are best handled by exec/CopyMem() or exec/CopyMemQuick() if they are patched with CPU specific optimized versions. The new vbcc versions of memcpy() will likely outperform unpatched exec/CopyMem() and exec/CopyMemQuick() for all sizes on the 68040 and 68060. Using an '=' for copies is likely to be the fastest for tiny to small copy sizes with practically all compilers on all 68k processors though.

I'm not clear which programs actually use CopyMemQuick() anyhow - probably the FFS does in case the "Mask" does not fit the target of the memory buffer (i.e. the handling device is "broken" and cannot reach the memory indicated as buffer by the caller). In any event, the impact CopyMem and friends has on performance of your average AmigaOs installation should be quite irrelevant. It would be interesting to see if someone could just set a breakpoint there and measure how often it is used. I suspend "not so often".

CopyMemQuick() is rarely use by the AmigaOS or any other programs (Scout is one of the few programs). CopyMem() is used extensively by the AmigaOS as well as many programs. The AmigaOS uses many tiny copies (<16 bytes) and it looks like some may be static. CopyMem() may have been used for tiny and small copies because the alignment is unknown and the source originally was for a 68000 where it could crash. It's fastest for tiny sizes to inline unaligned copies on the 68020+. I made a Snoopy (like SnoopDOS) script for CopyMem() and CopyMemQuick() to determine how often, which alignments, which sizes and which programs use these functions. The Snoopy script is available in this archive:

http://aminet.net/util/boot/CopyMem.lha

The script records the uses to memory. Be prepared to break the script after a few seconds from starting or you will run out of memory in less than a minute on the average Amiga. I wouldn't call that kind of use "not so often".

[guest11527]

I'm not so sure that "most" compilers will optimize memcpy() for static cases. Most compilers will optimize to some extent small copies implied with '=' which are practically all static sizes. The C memcpy() function is generally optimized for small to medium sized copies and doesn't detect for small static sizes. I have never seen the SAS/C memcpy()  function inlined (and it is poor for the 68040 and awful for the 68060).  

SAS/C certainly inlines memcpy() if you tell it to, but it will not "unroll" it if this is what you mean. I believe it expands into a simple move.b (a0)+,(a1)+ loop, likely non-ideal, but good enough for most cases.  

CopyMem() is used extensively by the AmigaOS as well as many programs. The AmigaOS uses many tiny copies (<16 bytes) and it looks like some may be static. CopyMem() may have been used for tiny and small copies because the alignment is unknown and the source originally was for a 68000 where it could crash. It's fastest for tiny sizes to inline unaligned copies on the 68020+. I made a Snoopy (like SnoopDOS) script for CopyMem() and CopyMemQuick() to determine how often, which alignments, which sizes and which programs use these functions.

Thus, out of curiousity, which programs actually call CopyMem() then after all? It's really nothing that *should* be called if a call can be avoided.

[psxphill]

I'm not clear which programs actually use CopyMemQuick() anyhow - probably the FFS does in case the "Mask" does not fit the target of the memory buffer (i.e. the handling device is "broken" and cannot reach the memory indicated as buffer by the caller).

The device isn't broken if it can't reach all memory. Exec was designed around not copying messages, therefore a device is broken if it does a CopyMem() to get the buffer into ZorroII or chip ram.
 
 It actually sounds like a hardware limitation to me.
 
     Mask=<0xffffffff>  Address Mask to specify memory range that DMA transfers can use at one time with any file system. Use Mask for compatibility with older hard drive systems.
 
 You should be able to achieve what you need with
 
     BufMem Type=<3>  Memory type used for buffers; (0 and 1 = Any, 2 and 3 = Chip, 4 and 5 = Fast).
 
 Although I don't know if you can specify 24 bit fast.
 
 It would make more sense when allocating memory to tell Exec what you were planning on doing with it, so it could allocate memory in the correct range and could also provide protection for it. This would go all the way back to the application that would allocate it's buffers saying it would be passing it to a dos stream, which would make sure the device could access it too.

[guest11527]

The device isn't broken if it can't reach all memory. Exec was designed around not copying messages, therefore a device is broken if it does a CopyMem() to get the buffer into ZorroII or chip ram.

Ok, let's be more precise: The *software implementation* is broken if it cannot satisfy the request of the user, or rather, overwrites other memory than the intended one because of a hardware limitation. In the end, the user (or caller) can hardly know all the limitations of the hardware, but the vendor of the hardware can. Thus, any good software implementation of an exec.device which does not know which memory it can reach or not reach, and cannot work around its limitations is broken.

The problem is: You cannot just open a device, perform a DoIO() or SendIO() and expect this to get the data. Now, where do you get the correct memory from to read data into? Answer is: The caller cannot possibly know that. Hence, "broken by design".

CBM fixed that for the trackdisk.device in 2.04. Good devices like the omniscsi are also aware of their limitations and use an additional IO buffer if necessary.

Anyhow, back to topic: Even though this is a possible application of CopyMem(), is that actually where it is used on your system?

[psxphill]

The problem is: You cannot just open a device, perform a DoIO() or SendIO() and expect this to get the data. Now, where do you get the correct memory from to read data into? Answer is: The caller cannot possibly know that. Hence, "broken by design".

The caller has to know or you'll get terrible performance, hence you provide that when you mount the file system. It's not perfect and should have been handled by asking the devices/libraries/file system what type of memory is acceptable.

By making it not work properly when given incompatible memory it at least forces the developer who is calling the device think about how to solve the problem in the most optimal way (which is likely by allowing the user to specify what type of ram to use as a buffer like the file system does).

[matthey]

SAS/C certainly inlines memcpy() if you tell it to, but it will not "unroll" it if this is what you mean. I believe it expands into a simple move.b (a0)+,(a1)+ loop, likely non-ideal, but good enough for most cases.

That is possible. I have seen many byte loops like this. It's o.k. for tiny dynamic sized copies only (small static sized copies should be unrolled MOVE instructions). It's not much more costly for small dynamic copies to align the copy and use a longword loop as the vbcc memcpy() inline does (with a byte copy for remaining and unaligned bytes). This is much faster for larger copy sizes at the cost of a little larger code.

Thus, out of curiousity, which programs actually call CopyMem() then after all? It's really nothing that *should* be called if a call can be avoided.

I believe the biggest use of exec/CopyMem() is the ram disk but I recall the graphics or maybe layers library using it as well. I'm out of town at the moment or I would run the SnoopDOS script and tell you.

[olsen]

I believe the biggest use of exec/CopyMem() is the ram disk but I recall the graphics or maybe layers library using it as well. I'm out of town at the moment or I would run the SnoopDOS script and tell you.

"ram-handler" may be among the most effective users of CopyMem(), maybe next to "scsi.device", but if you count the number of calls made to CopyMem() in the source code, "intuition.library" comes out on top. In ROM CopyMem() counts as a space saving measure, whereas on disk the operating system components happily use memcpy(), etc. instead.

[matthey]

"ram-handler" may be among the most effective users of CopyMem(), maybe next to "scsi.device", but if you count the number of calls made to CopyMem() in the source code, "intuition.library" comes out on top. In ROM CopyMem() counts as a space saving measure, whereas on disk the operating system components happily use memcpy(), etc. instead.

Yes, I believe there were intutition.library CopyMem() calls also. Maybe even 2nd or 3rd most frequent or most bytes copied. I can't verify or do any statistics right now. Almost all of the AmigaOS CopyMem() calls are less than 1kB with the average size much less. There are calls even when the computer is mostly idle.

[Heiroglyph]

I see both sides of the argument around here.

First and foremost, don't break the existing API. We don't get much new software, so the old stuff needs to continue working.

Realistically, we even need a few of the bugs if the original author won't or can't update the software to be compatible with fixed versions of the library. Obviously this needs to be evaluated on a case by case basis, but you get my point.

If there was actual movement by the software owners, then I'd be protecting them, but it's effectively, if not legally, abandoned.

If any of the owners still care, they really need to speak up and tell us who owns what, how to submit bugs and where to send money if needed. If you're inaccessible or invisible, don't complain when someone patches your code behind your back. Maybe someone should start a page for that kind of contact information.

If they don't care to accept bug reports and continue development, then I have no sympathy for their copyright protections. Morally I'm completely fine with whatever happens to abandoned code, including my own.

Beyond that, I'm all for what Cosmos is doing.

When you've got no C source, the disassembly is the only source available and he's making it readable.

While the speedups are good, assuming they translate to real world benchmark proven results, I think the most important and measurable improvement he's making is optimizing the size. We've only got so much ROM space so the more we can pack into there, the better. Removing C translation cruft is an easy way to reduce the size.

If he fixes some bugs without side effects, that's even better.

With the language barrier it's a little hard to tell if we're interpreting him correctly, but clearly his heart is in the right place and he's actually contributing to the community.

I applaud anyone who actually contributes, but damn we're an angry, uncooperative group of people.

[guest11527]

The caller has to know or you'll get terrible performance, hence you provide that when you mount the file system. It's not perfect and should have been handled by asking the devices/libraries/file system what type of memory is acceptable.

By making it not work properly when given incompatible memory it at least forces the developer who is calling the device think about how to solve the problem in the most optimal way (which is likely by allowing the user to specify what type of ram to use as a buffer like the file system does).

Ok, so let's think this to the very end, will we? If I call a device, I need to know the memory type it is able to handle and allocate that (is there a MEMF flag for every memory type a device needs? As for example, Zorro-II memory on *this* GVP board, or 32-bit memory on the CPU-board?). What if the caller needs the data in other memory regions? For example because it is an audio sample? Or a graphic for the graphics board? Or a program to load into 32-bit board memory? Thus, a caller has to create an off-target buffer, load the data in there and then copy it around.

What if the caller is the filing system? And the target is not in the right memory? Does the filing system has to do the copy? Or does the caller of the dos.library Read/Write routines also have to pass in the right buffer? What if want to read 5 bytes into a buffer I placed on the stack. Do I really need to allocate my own buffer for that and go through an AllocMem?

Or what happens if I want to copy data from device A to device B? Does the copy program have to know the memory requirements of all devices that participate in the copy? And create an off-target buffer in some cases, and not in other cases?  

Either way, you see where I'm getting at. You can't avoid the copy anyhow, and the best place to implement the copy is in the device, which knows *precisely* which type of memory it is able to handle, can obtain such memory by means besides exec (because it maintains the board it handles anyhow), and can perform the copy, in worst case, in the most efficient way since it can allocate the buffer exactly as needed.

Or to put it in a different way: There's a reason why every other operating system I know handles it the way I consider the correct way (off-side buffers for devices that cannot DMA into all regions), only AmigaOs doesn't. The FFS Mask/BufMemType flags are workarounds CBM invented because the trackdisk.device was broken and couldn't work with FFS out of the box. That's really all. Then it became an excuse for lazy device implementations, unfortunately.

[guest11527]

"ram-handler" may be among the most effective users of CopyMem(), maybe next to "scsi.device", but if you count the number of calls made to CopyMem() in the source code, "intuition.library" comes out on top. In ROM CopyMem() counts as a space saving measure, whereas on disk the operating system components happily use memcpy(), etc. instead.

Thanks, I forgot about RAM: Indeed, it has to copy memory. Strange that intuition uses it such a lot. Yes, I know CBM was tight on ROM space, but not *that* tight. All intuition does is copy a couple of small structures. In the end, a call to CopyMem() costs in best case four bytes (assuming that SysBase is already loaded), whereas a memcpy() costs six bytes (move.b (a0)+,(a1)+,subq.l#1,d0,bne loop). Wow, impressive...

Probably Green Hill C did not optmize this particular case.

[guest11527]

Realistically, we even need a few of the bugs if the original author won't or can't update the software to be compatible with fixed versions of the library. Obviously this needs to be evaluated on a case by case basis, but you get my point.

Or get new bugs because the new, ehem, author doesn't understand why particular choices were made, or at least, cannot read the sources...  

While the speedups are good, assuming they translate to real world benchmark proven results, I think the most important and measurable improvement he's making is optimizing the size. We've only got so much ROM space so the more we can pack into there, the better.  

No. Worse. Actually, most of the stuff should be *removed* from the ROM. Or are you suggesting that every time somebody makes an update users should burn a new ROM? RAM is cheap enough these days, even for Amiga. Loading the code into RAM costs nothing, it loads fast enough, and it is extremely simple to update it there. You can even write-protect the code in RAM if you want to. (MuProtectModules) So, there is absolutely no advantage of placing code in ROM.  

In the end, it's no improvement - the reverse engineered code does not loose its copyright just because it was reverse engineered, so it cannot be published, on a legal basis. Thus, whether anyone publishes the original source, or Cosmos puts his sources hack onto a public server is no difference. The difference is only *who* takes the risk, but not *why* or *if*. The problem remains the same problem, either way.

The only way to *solve* the problem is to make a clean-room implementation of the AmgiaOs code in first place. Yes, AROS does that. Thus, why not contribute there? Clean room implementation means: Person A will reverse engineer, will provide knowledge (but no code) to person B, who reimplements the interface as found from person A. To my very knowledge, this is a valid and legal approach to my very knowledge (but, IANAL). Actually, intuition interfaces are fully documented, thus not such a big deal. And if you want to ask people about internals, hey, you know where to look for.

One way or another, I think it is a completely superfluous undertaking, only fragmenting AmigaOs further, without actually offering the chance to resolve any single problem the platform has.

[wawrzon]

I see both sides of the argument

welcome back.

[wawrzon]

No. Worse. Actually, most of the stuff should be *removed* from the ROM.

im not exactly in accordance with this statement. yes, a lot of stuff might be moved between rom and disk, but dont forget amiga kickstart plays the role of bios initializing devices available at boot time and sicnce the setups are more various as the genuine it will be the question of flexibility and convenience to have drivers to access at least usb if not sata directly in rom. this as example, since i did that.

The only way to *solve* the problem is to make a clean-room implementation of the AmgiaOs code in first place. Yes, AROS does that. Thus, why not contribute there? Clean room implementation means: Person A will reverse engineer, will provide knowledge (but no code) to person B, who reimplements the interface as found from person A. To my very knowledge, this is a valid and legal approach to my very knowledge (but, IANAL). Actually, intuition interfaces are fully documented, thus not such a big deal. And if you want to ask people about internals, hey, you know where to look for.

exactly.

One way or another, I think it is a completely superfluous undertaking, only fragmenting AmigaOs further, without actually offering the chance to resolve any single problem the platform has.

not in my eyes, but if so, then its not the community fault. chances were there, for both sides, the enterprises and the consumers. if there was communication it could even actually take off for both. there wasnt. the community takes its path. sorry for those who missed the opportunity.