How to move AROS forward

[HenryCase]

@bloodline and @Piru
Thank you for your posts, I will now reply to them in one go.

bloodline wrote:
There is no way to know what the programs are passing to each other.

One thing at a time. First we need to know where the pointers are, then I'll say about my ideas on how we can use this information.

Piru wrote:
And I answer again: No.

Please explain the reasoning behind your answer, a simple 'No' doesn't help me understand why you don't think there is a way to see pointers in binary code.

Whilst I wait for your explanation, I would like to ask you another question. How does a CPU know that a program is asking to set up a pointer?

bloodline wrote:
No way, period. You could make assumptions but at best you would only be right, less than 30% of the time.

ASM is closely linked to the structure of machine code, correct? Also, ASM strives for efficiency. I would argue that there are only so many ways to set a pointer in ASM efficiently, and therefore only so many binary structures we would need to scan for to get a higher accuracy than 30% (where you got that figure from I don't know). I found a couple of ways of setting pointers in x86 ASM (I'm assuming it's x86 ASM anyway) from this page (Google cache version):
cs.colgate.edu/faculty/nevison/cs201web/lectureNotes/arraysandpointers.ppt
========================
clear2:
add  $t0, $a0, $zero    # p = addr of array[0]
forp:   sw   $zero, 0($t0)      # memory[p] = 0
addi $t0, $t0, 4        # p = p + 4
sll  $t1, $a1, 2        # $t1 = 4 * size
add  $t1, $t1, $a0      # $t1 = addr  array[size]
slt  $t2, $t0, $t1      # p < &array[size] ?
bne  $t2, $zero, forp   # if so, continue
endfor:
=========================
Pointer version assembly code
clear2:
add  $t0, $a0, $zero    # p = addr of array[0]
sll  $t1, $a1, 2        # $t1 = 4 * size
add  $t1, $t1, $a0      # $t1 = addr  array[size]
forp:   sw   $zero, 0($t0)      # memory[p] = 0
addi $t0, $t0, 4        # p = p + 4
slt  $t2, $t0, $t1      # p < &array[size] ?
bne  $t2, $zero, forp   # if so, continue
endfor:
========================
Why is it impossible to scan for the binary versions of the ASM code above?

[HenryCase]

@all
Thank you for your posts.

@bloodline and @SamuraiCrow
I would like to pick up on one point which has been stated by you both in the past, which is that there isn't a way to distinguish between instructions and data. In my admittedly limited understanding of the subject that seems like complete tosh, so forgive me whilst I persist with my questioning to discover who is mistaken.

According to Wikipedia, the following four steps are used in classical CPU architecture: fetch, decode, execute, writeback. The second step 'decode' is of interest to us here. Again, according to Wikipedia:

"In the decode step, the instruction is broken up into parts that have significance to other portions of the CPU. The way in which the numerical instruction value is interpreted is defined by the CPU's instruction set architecture(ISA). Often, one group of numbers in the instruction, called the opcode, indicates which operation to perform. The remaining parts of the number usually provide information required for that instruction, such as operands for an addition operation."

So for even the most basic program to function the CPU must know the difference between the opcode part of the instruction and the operand part of the instruction. Can we agree on that at least?

saimon69 wrote:
@HenryCase
I want to apologize to had pbrought the old discussion out, if only i knew was going to do such a mess, but maybe is better this way, so the ideas were put under verification.

No worries mate, I'm glad you did bring back this old discussion as it has been helpful for me.

saimon69 wrote:
i know through the developer mailing list that staf is rewriting the ABI

I took myself off the AROS mailing list, but I am curious to know how Staf is getting on with writing/rewriting the ABI. Any news you know of on that front?

[HenryCase]

yakumo9275 wrote:
You lack a basic fundamental understanding of how a cpu operates, so this thread goes around in circles when people say no, you say yes.

The idiot (like me) knows no limits, which is a strength as well as a weakness.

The cpu does know the difference between the opcode bits and other bits of the bytes it loads. The problem is that the cpu just loads a stream of bytes into its decoder. it looks at the bits and says mmm these 5 bits mean xor... I'll go run the XOR opcode. it doesnt know what its decoded is a string of text. It loads whatever the instruction pointer is pointing at. If things get really bad, it throws an exception.

"I'll go run the XOR opcode", so in other words I'm right.

The CPU has the ability to recognise an XOR opcode, and every other opcode it uses, and if part of an instruction isn't an opcode it is data. Therefore a CPU knows, after the 'decode' process, what is an opcode and what is an operand, correct? This requires just a yes or no answer.

[HenryCase]

yakumo9275 wrote:
Yes memory protection is so easy to add to amiga os

I never said it was easy, or even desirable, but I do think it can be done. What's wrong with wanting to discuss that?

Anyway the rest of your post before you went doolally was useful, so thank you.

[HenryCase]

@Hans
I'll reply to you soon, just off for some food.

@A6000

A6000 wrote:
When the CPU reads a memory location or a block of memory, it fetches something like this:-
1011001 11001100 11101010 00110110 01101111 00001110

the only way to interpret this "data" is to know what the CPU is expecting.

I think I have a way around this...

A6000 wrote:
So the only way for the operating system to know if a byte contains an opcode or data is to emulate the cpu in software and follow the execution of the program from the start, which will be very slow, it cannot simply read a byte and know whether it has fetched an opcode or data.

Bingo. CPU emulation is part of my plan. Yes it is too slow to use for running programs in real time, but it can be used to construct a 'map' of the program. This 'map' would only need to be created once, but would then be used by the operating system to help guide the program execution at key points.

[HenryCase]

@Hans

Hans wrote:
The problem is that your data may just happen to match an op-code. Op-codes are restricted in number, but data can take any value. Thus, if you recognize an XOR op-code in memory, it could be an op-code, but it might not.

Going back to something yakumo9275 mentioned in his last post:
"XOR is encoded with 5 bits, so after the 'decode' process, the 5bit XOR opcode is out, the processor is left with 26bits.. only they are not data, its still the same opcode. (all ppc instructions are 32bits)."

Do 68k processors have the same 'fixed length op-code' trait? If so, couldn't you read the first how ever many bits (16? 32?) after the program counter in the CPU incremented as the opcode?

Hans wrote:
You could also check whether the memory is marked as executable, in which case you could probably confirm that something is indeed an op-code.

That sounds promising.

Hans wrote:
Now what? That doesn't actually tell you what it's doing with data, who the data belongs to, etc.

Don't opcodes always have the same operand requirements ('x' many operand bits for one opcode format, 'y' many operand bits for another format)? If so, couldn't you use this format to determine what the data used by an operand represented in non-binary form?

[HenryCase]

@Hans_
Thanks for the quick reply.

Hans_ wrote:
No. You could perform an XOR on any type of data, from text, through to graphics, and even pointers.

Sorry, I worded my question badly, I meant more along the lines of the length of the data. So for instance certain addition instructions would have two blocks of data (i.e. the two values to be added together). Is there a way of determining the data length in terms of these blocks?

[HenryCase]

@pkillo
Thanks for your examples, I enjoyed following them. In your imaginary computer how does the CPU know when to increase the Program Counter?

@SamuraiCrow
I see. Well a fixed instruction width would have made our lives easier, but seeing as the vast majority of Amiga software is compiled for 68k CPUs let's see what we can do with that.

[HenryCase]

A6000 wrote:
One fly in the ointment for your idea of tracing program execution is that data may not be static, it might be input in real time to change program flow, or data might come from another part of the program which only executes if certain conditions are met, dependant on real world input, or data may come from another program entirely.
You may need to log all input data just in case the program you are tracing needs the data that came in during the trace time. it's all getting quite complicated.
You may not know the data found at the memory location is invalid.

A6000, this is the one issue I am trying to resolve at the moment. I'd much rather use some sort of feedback loop system that matches input, output and possibly the emulated CPU state, and analyse this information as a whole, rather than keep track of all inputs and work out the progression as a sequence.

I won't ask for 'any ideas' on this matter as I know I'll just get the usual 'it can't be done' stuff, and I'd much rather have a better explanation than that.

[HenryCase]

@A6000 and @SamuraiCrow

Thank you both for the 68k opcode information. My next question is linked to what you both said last. If 68k opcodes are 2-10 bytes in length, and one opcode can be directly followed by another opcode (i.e. no data in between) how does the instruction code tell the CPU to expect an opcode rather than operand data? Is there an opcode 'flag' instruction, or is it purely determined by the current opcode in use (and its operand needs)?

[HenryCase]

itix wrote:
You should try to disassemble some Amiga program and try to figure out where it is calling OpenWindow()... or figure out memory location where it stores struct Window.

Sounds like a good challenge.

[HenryCase]

@Piru
As I said before, I am currently looking of ways around this issue. The code map may end up being in a more high level modular structure. Maybe also we keep a record of the code paths we have taken before, and if a new one arises we track its performance, adding to our code map. These ideas are only rough sketches for now, I only realised through this thread that I'd have to deal with pointer issues when retrofitting Amiga operating systems and apps with MP, so forgive me for not going into more detail.

@all
Whilst I carry on sketching out how to track the performance of Amiga programs, I'd like to take the discussion about MP on a simpler route. Seeing as the whole idea of AROS being OS 3.1 compatible is on a source code level (apart from the plans for binary compatibility on the 68k port), let's start discussing how to graft full MP into AROS through program compilers. Would there be any way to make AllocMem (and all other AROS functions) behave as they should without using pointers (maybe using some new 'port' system instead that AROS could flag and control)?

[HenryCase]

@all
I'm at work so I don't have time to reply to all of your points.

@Trev

Trev wrote:
the Clay Institute will award you 1,000,000 USD for a generalized solution.

You get 1 million USD for solving that 400 students problem?! I've already started. ;-)

Piru wrote:
The code needs to be rewritten, compiler can't possibly know how to do that for you.

Yes it can. Let's say you rewrite the compiler to add a "-MPAROS" compiler flag. Then whenever you try and compile programs with this flag it follows certain rules to make programs compatible with an MP kernel. The resultant code is different from compiling it normally, but crucially you wouldn't need to rewrite the application code yourself.

[HenryCase]

@bloodline and @Hans_
Thank you both again for informative posts. I think you misunderstand my plan, but that's understandable as I haven't explained it fully. Think of it like API translation at the compiler level. So instead of declaring AllocMem() as a pointer, the same attributes about the memory requirements (memory size, flags) get declared in a malloc kind of way. What I aim to do is take the API calls and turn them into ones that can support MP. Does this sound more reasonable?

@koaftder
Cool. I'll check out your VM when I get time ("99 bottles of beer on the wall, 99 bottles of beer...").

[HenryCase]

Hans_ wrote:
In theory it's all possible simply because we know that humans can do it.

I think that's one of the factors that is driving me on.

bloodline wrote:
In an MP OS, one would define that the pointer returned by AllocMem() is for the use of the calling task only.
In AmigaOS the pointer returned by AllocMem() is free for any task to use, this is defined behaviour.

Okay, so Amiga software expects to be able to use pointers defined by other Amiga software. So there is almost a pool of memory pointers. Can we put these memory pointers in an OS controlled environment and control which programs we'd like to use these pointers?

In the place of the pointers in the application code we put a 'port' to the shared pointer area. When a program then wants to access the data or instruction in memory a request to the OS goes to through that port. The OS then uses the pointer itself, and the outcome of this action is stored in a 'quarantine' area in memory. The OS then tries to determine (based on past experience and/or user preference) whether that data should be given to the application.

Way I see it, if a program causes a computer to crash, we should be able to determine when it crashed, and what was the last value (data or instruction) to be passed for processing, right? If we can do this, we have options when it comes to crash protection (which is the main benefit of MP IMO), as long as we catch the value before it is passed for execution (or maybe before storage in its designated memory location).

Or maybe we don't have those options... can you give me typical low level examples of what causes a computer to crash?

Hans_ wrote:
What exactly is wrong with allowing old software to be fully-allocated in shared-memory and run in a sand-box?

I want to see the AmigaOS 3.1 API succeed as much as possible as I see it as a bridge between different Amiga-based OSs. A sandbox is okay, but it would split the community again as there would be a new 'native' API. AROS is in the unique position to bring the Amiga world together, and I don't want to see it lose that.

Hans_ wrote:
Henry is suggesting getting the compiler to insert the MEMF_SHARED flag (and/or whatever else woulld be used) whenever it's necessary. The problem is that finding out when it's necessary is incredibly complex.

Correct, I'm advocating making memory private by default. The 'finding out when sharing is neccessary' task I am trying to pass to humans.

bloodline wrote:
He needs to understand that the messages and message passing are not the problem, it's the contents that's the problem.

So I would prefer it if we could keep a examples as simple as possible.

bloodline, I appreciate your approach here. I agree it is best to keep the information you are all giving me as simple as possible, I do feel like I am slowly getting my head around the larger issues here, thank you for your patience (and everyone elses). The thing is I'm finding this discussion rather interesting, and as I'm getting more information it's throwing up more questions and possibilities in my head. Please forgive/understand my curiosity!

[HenryCase]

itix wrote:
Again everything which is passed between applications and kernel must be public memory. Using private memory in Amiga is useless because such memory can not be passed to kernel calls because you dont know what kernel does with it. You can not trust that library calls can handle private memory.

So in an MP protected OS like, for example, FreeBSD, how do instructions/data get passed between applications and kernel?