Any OS legal way to traverse an exec memory pool?

[Karlos]

Let's assume you are using CreatePool() / DeletePool() to manage memory allocation. If you wanted to traverse the allocations made so far so that you could, for example write to a debug log, is there any OS legal way to access the memory list?

The problem is that, according to the includes&autodocs, CreatePool() returns void* and DeletePool() takes void*, which tends to suggest the underlying implementation is completely private and I shouldn't assume anything about what is actually pointed to.

I can implement my own tracker easily enough if there is no sanctioned way of getting the information but that would seem like reinventing the wheel when it's clear a memory pool already tracks the allocations made within it.

[AJCopland]

Wouldn't it be better to write your own memory manager as OS calls for memory allocation are generally fairly slow. Though I admit that might not be as true for the Amiga since I haven't coded for it in so long.

Andy

[Karlos]

Any allocator you write is going to have to ask the OS for memory at some point. You can allocate large blocks from the OS infrequently and manage smaller allocations yourself from within those. In theory, the memory pool system was supposed to do away with the need for this.

[AJCopland]

Ah right, on memory limited machines (i.e. consoles) I generally use a homebrew dynamic allocator that just takes the entire lot right at startup and breaks it into chunks with push n' pop style functionality.

Seems like the same idea but you'd have control over everything as well as logging and debugging info.

Andy

[Piru]

Let's assume you are using CreatePool() / DeletePool() to manage memory allocation. If you wanted to traverse the allocations made so far so that you could, for example write to a debug log, is there any OS legal way to access the memory list?

No.

[Karlos]

But... How can I be sure you are the real Piru? :-?

:lol: Actually, scrub that, only the real Piru would be this terse:

No.

As it happens, I am wrapping the pool and I have a tiny header for each allocation (I never use it for trivial allocation of a few bytes anyway), so as far as I can see there's no reason I couldn't add basic prev / next node pointers in there.

[CrazyProg]

If you are just interested in a debug log, you could use IExec->SetFunction() :rtfm:  to put in a couple of functions that print the args, calls the orginal function, then prints the results, for both IExec->AllocPooled() and IExec->FreePooled().

That way you don't need to know the internal layout. :-D

I personally haven't used SetFunction() on OS4, by the description it seems to be more relevent to pre OS4 libaries.  

There is a hack you can use instead.
  1) carefully write you function according to the prototypes  in the interfaces drawer.
  2) open the library (not needed for exec)
  3) remember what the vector is (oldvector = IExec->AllocPooled;)
  3) Set the vector for the function (IExec->AllocPooled = MyAllocPooled;)
  4) do your stuff
  5) restore the function pointer back to what is was (IExec->AllocPooled = oldvector;)
  6) exit your program.


Remember that anything at anytime can call your new vector you might need to use a semaphore to make sure you prints don't get jumbled up.

[Piru]

@CrazyProg

Uh, if it's your program calling the functions why the heck would you want to patch exec?

Karlos already said it:

As it happens, I am wrapping the pool and I have a tiny header for each allocation (I never use it for trivial allocation of a few bytes anyway), so as far as I can see there's no reason I couldn't add basic prev / next node pointers in there.

And this is exactly what he should do.

But the original question, if there is a OS legal way to traverse the allocated memory in a pool. There isn't (for the obvious reason: Memory pool is a "black box", it's internal implementation can change at any time, and has in the past already. AmigaOS 3.9 BB2 introduced new binary tree memory pools).

Anyway, there is a fatal flaw in patching the system in a way you call the original function, and then examine the result: There is no way to be 100% sure the function has returned. In this case, even if you remove the patch the code could still be executing, and your function could be landed long after the executable has been unloaded by shell or workbench. Adding some call counter won't help either, as you could still get rescheduler after you subtract the counter and before executing the final 'return from subroutine'. As such, there is no safe way to remove patches like this.

[Karlos]

So, it looks like I have to do my own tracking.

In the allocation system, so far, every allocation has a (private) 16-byte header that immediately precedes the allocated data. There may be some waste data before the header due to cache alignment code:

Code: [Select]


struct MemAllocation {
  uint32 identifier;      // used to identify this as an allocation from our system
  uint32 allocationSize;
  uint32 allocationFlags;
  void*  realAddress;     // the real base address that was returned from AllocPooled()
};



The allocation routine aligns the allocation to a cacheline boundary, so this structure is cache aligned, as is the space following it.

The identifier is a unique ID that allows the corresponding free method to ascertain if it is legal to release the memory pointed to. Basically if the uint32 value 16 bytes before the returned address (having done the obvious null checks first) matches this id, it is a block you can free. Of course, the free routine zeroes out this field so that if you ever did try to free it twice, the second free will fail gracefully.

I guess I can extend this and add MemAllocation* prev and MemAllocation* next, but that leaves a further 8 bytes to fill with useful data or rejig the alignment code. I suppose a 32-64 bytes overhead (inclusive of alignment padding) per allocation isn't so much these days.

Besides, I could always conditinally compile them in/out once the entire thing is sufficiently debugged.

[Piru]

@Karlos

Sounds like a plan. The similar system I implemented also had walls (that is some running pattern before and after the allocated area). Af memory free I'd check for wall hits and report them.

Also, PoolWatch could be useful in some cases.

[Doobrey]

CrazyProg wrote:

There is a hack you can use instead.
  1) carefully write you function according to the prototypes  in the interfaces drawer.
  2) open the library (not needed for exec)
  3) remember what the vector is (oldvector = IExec->AllocPooled;)
  3) Set the vector for the function (IExec->AllocPooled = MyAllocPooled;)
  4) do your stuff
  5) restore the function pointer back to what is was (IExec->AllocPooled = oldvector;)
  6) exit your program.

If you're gonna do it that way, add a check to step 5 to only restore the vector if it still points to your function.If it doesn't match then it means something else has SetFunction()'d it too and could possibly call your old function, with no way of knowing you've removed it.

[Homer]

Sooo, whats this thread about then guys  :lol:

[Karlos]

There's no way I'd consider patching the OS just to implement a feature for my own libraries. I think simply extending my header structure to include a pair of nodes and using the remaining bytes prior to the allocation base address as a wall (and some more afterwards) that can be removed once debugging is complete would be more than sufficient.

Having said that, I haven't had an out of bounds pointer bug in anything I've written for years ;-)