Welcome, Guest. Please login or register.

Author Topic: Code compatibility to memory protection model on various platforms etc.  (Read 2530 times)

Description:

0 Members and 1 Guest are viewing this topic.

Offline JoseTopic starter

  • Hero Member
  • *****
  • Join Date: Feb 2002
  • Posts: 2871
    • Show only replies by Jose
Ok, this one is for Piru, or other experts out here:)

The bellow questions is in the context of the library I'm doing, which streams any data very easily. Appliable to all the many situations where serialization is usable (saving to HD, sending over the network etc,
actually those are the two only likely uses that have occurred to me regarding this code...).
You've probably seen a few posts of me mentioning it and Yesssss! I finally got the macros to a usable state;) :) More
about them latter, I think you'll be surprised, I'm dead happy with what I've achieved and I've already got a few
ways to improve them even further, but thing is they're pretty usabble now and if I do more C macro coding I'll start to
through up (really...).

 Anyway....:) my idea for the library code, to make the thing rule, would be to have a circular buffer, to wich the library serializes the data and whose contents in turn can be streamed in smaller chunks, so that the library can continue to serialize data to other buffer's chunks, while one is being streamed at the same time in a piece big enouph to not hit performance. The streaming would happend a bit like the RawDoFmt() amiga rom function, with each library call that asks for serialization giving a pointer to a function that is called in a loop through the various chunks, wich does whatever the caller wants to do with it (save to hd, send through network...). The difference here is that RawDoFmt passed a char at a time to function, mine would pass a chunk to be much more performant. For this to work, the passed function has to be called in a separate library process that receives messages from the serializing library process, and that marks each chunk as being processed or not, after the function has returned.
So basically when the serialization reaches the chunk being used by the passed function the buffer becomes full and it waits for current function call to finish. Otherwise the data is being serialized to the buffer as it's contents are being used at the very same time.

Questions:
1- Generally, in most platforms, the library code is executed in the caller's memory space right ? Meaning that the separate library process can call passed function without any problems, otherwise the function could be accessing forbidden memory space for it. But in this case, the thing gets complicated because the library call sends a message to the process the calls the fucntion, so it might not work.
2- Do other platforms allow this easily ? I've done something a bit similar in the past with ports and signals and it was straightforward but I've done it on the Amiga (no memory protection). This one I want to be multiplatform, wich I think will probably be a big PITA...
3- What do you guys think of the model, do you advise any improvements ?

:pint: :pint: :pint:
Jose
\\"We made Amiga, they {bleep}ed it up\\"
 

Offline JoseTopic starter

  • Hero Member
  • *****
  • Join Date: Feb 2002
  • Posts: 2871
    • Show only replies by Jose
Re: Code compatibility to memory protection model on various platforms etc.
« Reply #1 on: August 03, 2007, 02:38:19 PM »
Hey wake up, time to heal that hangover with some coding time !!!
Or some :pint:
\\"We made Amiga, they {bleep}ed it up\\"
 

Offline MskoDestny

  • Sr. Member
  • ****
  • Join Date: Oct 2004
  • Posts: 363
    • Show only replies by MskoDestny
    • http://www.retrodev.com
Re: Code compatibility to memory protection model on various platforms etc.
« Reply #2 on: August 03, 2007, 08:00:20 PM »
Quote

Jose wrote:
1- Generally, in most platforms, the library code is executed in the caller's memory space right ?

Every system I'm familiar with.

Quote
But in this case, the thing gets complicated because the library call sends a message to the process the calls the fucntion, so it might not work.

Why can't the receiver of the message just be another thread within the same process?

Quote
2- Do other platforms allow this easily ? I've done something a bit similar in the past with ports and signals and it was straightforward but I've done it on the Amiga (no memory protection). This one I want to be multiplatform, wich I think will probably be a big PITA...

AFAIK, all the major platforms that support memory protection also have some kind of mechanism for creating a region of memory that is shared between multiple processes. However, I believe the mechanism varies somewhat from platform to platform.
 

Offline JoseTopic starter

  • Hero Member
  • *****
  • Join Date: Feb 2002
  • Posts: 2871
    • Show only replies by Jose
Re: Code compatibility to memory protection model on various platforms etc.
« Reply #3 on: August 03, 2007, 08:24:42 PM »
@MskoDestny
"Why can't the receiver of the message just be another thread within the same process?"

Because I'd like library usage to be as simple as calling any other function, without the need to create a message port, receive it's messages, processing them, and clean up everything in the end.

"AFAIK, all the major platforms that support memory protection also have some kind of mechanism for creating a region of memory that is shared between multiple processes. However, I believe the mechanism varies somewhat from platform to platform."

As long as it's possible it's a big thing in this case, cause I still haven't thought of another way to implement it.
\\"We made Amiga, they {bleep}ed it up\\"
 

Offline MskoDestny

  • Sr. Member
  • ****
  • Join Date: Oct 2004
  • Posts: 363
    • Show only replies by MskoDestny
    • http://www.retrodev.com
Re: Code compatibility to memory protection model on various platforms etc.
« Reply #4 on: August 03, 2007, 08:57:27 PM »
Quote

Jose wrote:
@MskoDestny
"Why can't the receiver of the message just be another thread within the same process?"

Because I'd like library usage to be as simple as calling any other function, without the need to create a message port, receive it's messages, processing them, and clean up everything in the end.

I'm not sure I follow. In both cases there is need for setup and cleanup. With a separate process you need to spawn the separate process and allocate a shared memory area. The user of the library will need to stick his buffers in the shared memory area if you want to avoid copying it. When you're done with the library the extra process needs to be closed and the shared memory freed.

With a separate thread, you need to spawn the thread and setup the mechanism by which you're going to send messages to it. There are no special memory allocation requirements put on the user of the library. When you're done you kill the thread and clean up your message mechanism.

Also, I don't see why you need a full OS message port. Just write a really simple message queue using some simple data structure (linked list perhaps) and a mutex to control access.

Of course, I should say I know next to nothing about programming on Amiga OS so perhaps I'm missing something here.
 

Offline JoseTopic starter

  • Hero Member
  • *****
  • Join Date: Feb 2002
  • Posts: 2871
    • Show only replies by Jose
Re: Code compatibility to memory protection model on various platforms etc.
« Reply #5 on: August 03, 2007, 09:52:16 PM »
Wait, you're right about the caller not needing to create a port, I misinterpreted your previous post.

Regarding your process vs. thread examples:
With a process I was thinking about the other way around, the library allocating the buffers on user call (which would request a specific size or just the default value). The function passed by the user to act on the buffer would be the problem, because sometimes the caller might want that function to write a part to / include something in it's own memory area (the caller would also specify void * pointer to any data (in this case in it's own area) along with the function to call).
This is more or less like RawDoFmt() works.

If the library creates a thread within the same memory space as the caller then I guess there would be no problem :)
\\"We made Amiga, they {bleep}ed it up\\"
 

Offline JoseTopic starter

  • Hero Member
  • *****
  • Join Date: Feb 2002
  • Posts: 2871
    • Show only replies by Jose
Re: Code compatibility to memory protection model on various platforms etc.
« Reply #6 on: August 03, 2007, 09:58:48 PM »
BTW, regarding AmigaOS, the classic version doesn't have memory protection at all, AOS4 and MOS do have it, though it's use is optional on the application side.
But I want the library to be multiplatform (Linux, Mac, Windows, BeOs, OpenBSD ....)
\\"We made Amiga, they {bleep}ed it up\\"