Welcome, Guest. Please login or register.

Author Topic: TLSFMem O(1) Memory Allocator  (Read 4236 times)

Description:

0 Members and 1 Guest are viewing this topic.

Offline platon42Topic starter

  • Hero Member
  • *****
  • Join Date: Jul 2002
  • Posts: 573
    • Show only replies by platon42
    • http://www.platon42.de/
Re: TLSFMem O(1) Memory Allocator
« Reply #14 on: October 15, 2007, 07:50:00 PM »
> Hey, do you think that this bug could explain the (urg!..) partition trashing experiences that i had with PoolMem and AllocP?? (No problems with MemOptimizer, so far)
> What are the expected consequences of this? I do know that you provide a bug-fix, but i still would like to know.

Could be a result indeed. Whatever application gets access to this memory by allocating it first and using it could cause all kind of weird behaviour -- and as the important pointers (io_Data, io_Offset and io_Length) are in the back part of the structure, partition trashing could occur.

> Oh, another (important, i believe) question please: Many ppl like me are using the IDEFix package which patches/replaces the standard scsi.device. What's the case with this one? Is there a need for a fix? Which one from the ScsiBugfix directory?

I don't think that the idefix drivers are derivates from the Commodore ones, hence they should not have this bug.
--
Regards, Chris Hodges )-> http://www.platon42.de <-(
hackerkey://v4sw7CJS$hw6/7ln6pr7+8AOP$ck0ma8u2LMw1/4Xm5l3i5TJCOTextPad/e7t2BDMNb7GHLen5a34s5IMr1g3/5ACM
 

Offline platon42Topic starter

  • Hero Member
  • *****
  • Join Date: Jul 2002
  • Posts: 573
    • Show only replies by platon42
    • http://www.platon42.de/
Re: TLSFMem O(1) Memory Allocator
« Reply #15 on: October 15, 2007, 07:55:17 PM »
> Forgive me for being a bit dense, but I don't quite get it. What does this do to my system? Would I be correct in thinking that it makes memory access faster? Significantly?

It doesn't speed up memory access -- this is hardware bound. But programs that use the exec memory functions for reserving and freeing dynamic memory will benefit from the new routines. Significantly.

For example, when I run my backup script on my harddisk, which will use lha to scan a partition with >80000 files, it will create over 5000 memory fragments of 8 bytes size. This sucks and makes my system *crawl*. In the past, I just called AllocFrags at regular intervals which will remove the 8 byte fragments, but after the backup process, my memory is a piece of swiss cheese (and lha needs about a minute to release its memory).

With TLSFMem, this fragmentation is gone and will speed up these tenthousands of memory operations significantly.
--
Regards, Chris Hodges )-> http://www.platon42.de <-(
hackerkey://v4sw7CJS$hw6/7ln6pr7+8AOP$ck0ma8u2LMw1/4Xm5l3i5TJCOTextPad/e7t2BDMNb7GHLen5a34s5IMr1g3/5ACM
 

Offline AmigaMance

  • Hero Member
  • *****
  • Join Date: Apr 2005
  • Posts: 1278
    • Show only replies by AmigaMance
Re: TLSFMem O(1) Memory Allocator
« Reply #16 on: October 15, 2007, 08:26:09 PM »
@platon42
 Thanks for the reply.
 I have found a serious problem with your patch:
Every WOS and PUP progs i have tried so far, refuses to work when TLSFMem is installed. :-(

 Anyone else have this problem?
A1200 PPC user.
 

Offline motorollin

  • Hero Member
  • *****
  • Join Date: Nov 2005
  • Posts: 8669
    • Show only replies by motorollin
Re: TLSFMem O(1) Memory Allocator
« Reply #17 on: October 15, 2007, 08:43:56 PM »
Sounds great! Thanks for this. So would a good test of performance increase be to time how long it takes to create a huge lha file, then use your patch and do the same thing again?

--
moto
Code: [Select]
10  IT\'S THE FINAL COUNTDOWN
20  FOR C = 1 TO 2
30     DA-NA-NAAAA-NAAAA DA-NA-NA-NA-NAAAA
40     DA-NA-NAAAA-NAAAA DA-NA-NA-NA-NA-NA-NAAAAA
50  NEXT C
60  NA-NA-NAAAA
70  NA-NA NA-NA-NA-NA-NAAAA NAAA-NAAAAAAAAAAA
80  GOTO 10
 

Offline cybernoid

  • Newbie
  • *
  • Join Date: Jun 2006
  • Posts: 34
    • Show only replies by cybernoid
Re: TLSFMem O(1) Memory Allocator
« Reply #18 on: October 16, 2007, 02:28:25 AM »
On my A4000 25 standard (+dvd & masobochi 8mg ram card) i get those promising:
sysspeed vs Ak4_040

LhaCrunch 6.36 (7.95)
LhaTest (=)
LhaDeCrunch 0.76 (1.10)
XPK 14.48 (15.52)
XPKDe 2.14 (2.18)
PPCrunch 13.62 (13.96)
PPDe 0.50 (0.80)

Yes, my system was already optimized, but even optimized I experienced gain of about 10%.

I also notice a VERY BIG gain in the intuition library, special here:

movewin16 40 (18) ((35)
movewin256 27 (3) ((17)) !!! UAU !!!

So, this is GREAT.  :rtfm:
 

Offline dammy

  • Hero Member
  • *****
  • Join Date: Nov 2002
  • Posts: 2828
    • Show only replies by dammy
Re: TLSFMem O(1) Memory Allocator
« Reply #19 on: October 16, 2007, 09:35:36 AM »
Are the sources included in the archive?

Dammy
Dammy

https://www.facebook.com/pages/Arix-OS/414578091930728
Unless otherwise noted, I speak only for myself.
 

Offline JosephC

  • Jr. Member
  • **
  • Join Date: Aug 2007
  • Posts: 63
    • Show only replies by JosephC
Re: TLSFMem O(1) Memory Allocator
« Reply #20 on: October 16, 2007, 11:21:13 AM »
I want this patch to become the standard.
But I don't see how that will happen when it can't be run with debugging software such as MuGuardianAngel.

What can we do to make MuGuardianAngel cooperate with TSLFMem?

Please someone must have the answer!
 

Offline ChrisH

  • Jr. Member
  • **
  • Join Date: Jun 2007
  • Posts: 92
  • Country: 00
    • Show only replies by ChrisH
    • http://cshandley.co.uk/email
Re: TLSFMem O(1) Memory Allocator
« Reply #21 on: October 16, 2007, 11:57:55 AM »
@platon42
Quote
Regarding the "inefficiency" of 103% is about the good fit properties, it
doesn't waste memory (but I don't know about the OS4 allocator).

But the "good fit properties" of 103% means that it uses (up to) 3% more memory than necessary, so it *is* an inefficiency.  OS4 has a similar wastage of (up to) 12.5%.  BTW, the proper name for this is "internal fragmentation", unless I misunderstand you.
Author of the PortablE programming language.
It is pitch black. You are likely to be eaten by a grue.
 

Offline ChrisH

  • Jr. Member
  • **
  • Join Date: Jun 2007
  • Posts: 92
  • Country: 00
    • Show only replies by ChrisH
    • http://cshandley.co.uk/email
Re: TLSFMem O(1) Memory Allocator
« Reply #22 on: October 16, 2007, 12:03:41 PM »
@motorollin
Quote
Forgive me for being a bit dense, but I don't quite get it. What does this do to my system?

For some programs it will make NO difference, but for those that allocate lots of (usually small) chunks of memory, it could make a massive difference:

For my own (very old) FolderSync program, I was able to gain a 70 times speed-up by writing my own custom memory allocation system.  The reason is that (a) AmigaOS 3.x has a very old & slow memory allocation system, and (b) FolderSync made very many (small) memory allocations.
Author of the PortablE programming language.
It is pitch black. You are likely to be eaten by a grue.
 

Offline AmigaMance

  • Hero Member
  • *****
  • Join Date: Apr 2005
  • Posts: 1278
    • Show only replies by AmigaMance
Re: TLSFMem O(1) Memory Allocator
« Reply #23 on: October 16, 2007, 04:12:34 PM »
Quote
I have found a serious problem with your patch:
Every WOS and PUP progs i have tried so far, refuses to work when TLSFMem is installed.

 So..  Is there anyone with a PPC board who confirms this?
A1200 PPC user.
 

Offline RW222

  • Full Member
  • ***
  • Join Date: Oct 2007
  • Posts: 155
    • Show only replies by RW222
Re: TLSFMem O(1) Memory Allocator
« Reply #24 on: October 16, 2007, 05:14:09 PM »
Interesting, last time I was using my A1200 heavily I'd have to keep running a util to defrag and reclaim the RAM, chip RAM in particular used to lose 200K-400k or so pretty quickly. Think the util I was using was called "memory hogs" or something like that.


Now if you want a perfect memory allocation system, howabout if it was divided up into 64KB banks, and each program was given it's own 64KB bank. If you had 10 banks that would be 640K which ought to be enough for anyone..  :-P

(Just kidding of course)
RW222: A1200 (early commodore) A1220 Turbo+4MB, A500x2.
 

Offline nOw2

  • Full Member
  • ***
  • Join Date: Jul 2002
  • Posts: 194
    • Show only replies by nOw2
Re: TLSFMem O(1) Memory Allocator
« Reply #25 on: October 16, 2007, 07:37:03 PM »
This sounds like one of those rare bits of brilliance.
I'm keen to get this going as the only reason my A4000/060/CS-Mk3 reboots these days is when memory fragmentation slows it down after 20-30 days uptime.

I can get it to run by booting with no startup, but tools like VirusZ report bus errors. I could live with that, but later in the bootup something on loading Opus5 causes a guru. I can't see anything significant in WBStartup.

Has anyone encountered this? Is there somewhere else I should go for support where I  can post better diagnostics?

My best guess, since the buserrors start after SetPatch, is perhaps it doesn't like this (I can't find a copy of the Phase5 libs just now to test):
> version libs:68060.library full
68060.library 40.18 (12/11/2001)
(c) 1999-2001 The MMU.lib development group, THOR
 

Offline platon42Topic starter

  • Hero Member
  • *****
  • Join Date: Jul 2002
  • Posts: 573
    • Show only replies by platon42
    • http://www.platon42.de/
Re: TLSFMem O(1) Memory Allocator
« Reply #26 on: October 17, 2007, 08:29:19 AM »
> Are the sources included in the archive?

No. I don't think that ~2000 lines of 68k-asm would be helpful for AROS either. There is that paper and at least two open source implementations of TLSF in C on the internet.
--
Regards, Chris Hodges )-> http://www.platon42.de <-(
hackerkey://v4sw7CJS$hw6/7ln6pr7+8AOP$ck0ma8u2LMw1/4Xm5l3i5TJCOTextPad/e7t2BDMNb7GHLen5a34s5IMr1g3/5ACM
 

Offline platon42Topic starter

  • Hero Member
  • *****
  • Join Date: Jul 2002
  • Posts: 573
    • Show only replies by platon42
    • http://www.platon42.de/
Re: TLSFMem O(1) Memory Allocator
« Reply #27 on: October 17, 2007, 08:34:00 AM »
> But I don't see how that will happen when it can't be run with debugging software such as MuGuardianAngel.

Well, developers will usually have their "debugging setup". I'm not running MuGuardianAngel during the normal system operation :)

> What can we do to make MuGuardianAngel cooperate with TSLFMem?

MuGuardianAngel *replaces* the exec.library functions, just as TLSFMem does. But as it does, it will bork because the old memory headers are nearly empty (thus no free memory) and a free memory check on the TLSF memory headers will cause a (recoverable) guru, because it doesn't contain any chunks, but the MH_FREE field is not 0. Also I would expect MuGuardianAngel to return MMU page aligned allocations to protect them from harm -- this is currently not supported in TLSFMem.
--
Regards, Chris Hodges )-> http://www.platon42.de <-(
hackerkey://v4sw7CJS$hw6/7ln6pr7+8AOP$ck0ma8u2LMw1/4Xm5l3i5TJCOTextPad/e7t2BDMNb7GHLen5a34s5IMr1g3/5ACM
 

Offline platon42Topic starter

  • Hero Member
  • *****
  • Join Date: Jul 2002
  • Posts: 573
    • Show only replies by platon42
    • http://www.platon42.de/
Re: TLSFMem O(1) Memory Allocator
« Reply #28 on: October 17, 2007, 08:50:39 AM »
Quote

But the "good fit properties" of 103% means that it uses (up to) 3% more memory than necessary, so it *is* an inefficiency. OS4 has a similar wastage of (up to) 12.5%. BTW, the proper name for this is "internal fragmentation", unless I misunderstand you.


No, thats exactly NOT what it means. Good fit means that *if* there is a memory block of at least that size category, it will find it in O(1) -- otherwise the next bigger block is returned (also in O(1)). If the sizes are exactly the same, it will allocate and eat it, if the free block was bigger, it will split it accordingly AND NO MEMORY IS WASTED. The 103% only means that a "size category" is between 100% and 103% of the requested size, and it will not distinguish between a set of different free blocks within a category. I suggest to have a look at the paper describing TLSF.

Good fit means, that it will find a free block that will have a size /close/ to the requested block. Best/Exact fit would mean, the algorithm had to search *all* the free memory chunks for a block that fits exactly.

In any case, if there's only bigger chunk of free memory than requested, the memory block is split. And the remaining block is still available to the system. No memory is wasted.

Thus TLSFMem does not have internal fragmentation except for its alignment and allocation sizes need to be multiples of 16. If the OS4 allocator has internal fragmentation and if it's really up to 12,5%, it sucks.
--
Regards, Chris Hodges )-> http://www.platon42.de <-(
hackerkey://v4sw7CJS$hw6/7ln6pr7+8AOP$ck0ma8u2LMw1/4Xm5l3i5TJCOTextPad/e7t2BDMNb7GHLen5a34s5IMr1g3/5ACM
 

Offline platon42Topic starter

  • Hero Member
  • *****
  • Join Date: Jul 2002
  • Posts: 573
    • Show only replies by platon42
    • http://www.platon42.de/
Re: TLSFMem O(1) Memory Allocator
« Reply #29 from previous page: October 17, 2007, 08:56:05 AM »
Quote

For my own (very old) FolderSync program, I was able to gain a 70 times speed-up by writing my own custom memory allocation system. The reason is that (a) AmigaOS 3.x has a very old & slow memory allocation system, and (b) FolderSync made very many (small) memory allocations.


Isn't that what memory pools, introduced with OS 2.0 is for? Small allocations, pooled into puddles? Unfortunately, there are still programs, that don't use them, way past the OS 2.0 release date...
--
Regards, Chris Hodges )-> http://www.platon42.de <-(
hackerkey://v4sw7CJS$hw6/7ln6pr7+8AOP$ck0ma8u2LMw1/4Xm5l3i5TJCOTextPad/e7t2BDMNb7GHLen5a34s5IMr1g3/5ACM