Amiga.org
Amiga computer related discussion => Amiga community support ideas => Topic started by: uncharted on July 24, 2008, 09:59:26 PM
-
I'm not going to beat about the bush here, as much as I appreciate the work that has gone into AROS, and admire the dedication of those behind it, AROS has become stagnant and is frankly almost as useless as it was 5 or 6 years ago. Sure there has been progress, but that progress has been directionless and of no benefit for most of the Amiga community.
I'm not posting here to beat on the AROS developers. I'm here to be constructive and put forward a proposal of how things could be resolved. However, the first obstacle in the way of anything happening is to get the core AROS guys to acknowledge that there is a problem, and that THEY need to do something about it. That is why I have to be so blunt in the post about the dire state things are in. AROS is currently stuck between a rock and a hard place. The rock is the lack of binary compatibility and the hard place is the restrictions imposed by source compatibility. Hiding behind the APL just isn't going to cut it anymore.
My proposal is simple: Fork the AROS project into 2 distinct projects, each with a different technical focus and target audience. For the purposes for distinction, I'll refer to them as 'Classic' and 'Future' here. This is by no means my suggestion for naming.
Let's start with Classic. The Classic project's main aim would be 3.1 binary compatibility. The primary target platform would be *UAE, with real physical 68k machines the secondary target. Strip out anything that isn't needed to run in UAE - drivers etc. There are plenty of improvements that Classic could provide over OS3.1. It could be then used as a base for distribution builders such as AmigaSYS, AmiKit and ClassicWB to add value to.
With the Classic project handling the compatibility side of things, the Future project can concentrate on making something a bit more modern. No more skirting around Memory protection with small bits here and there. Full MP is now possible. All those 1980's restrictions are lifted, and the developers can concentrate on making something Amiga-like rather than Amiga compatible. Compatibilty might be added later through a sandbox technique utilising Classic as a hosted OS or even simply using classic under E-UAE, which seems to be the direction the core AROS team have favoured before.
Freeing Future should hopefully spark a bit more interest from outside the community. It appeared to me that one of the reasons certain Devs have left the project is that the Amiga restrictions have prevented them from building the OS they wanted.
Before going into a big more detail, I just want to be honest here. In this scheme of things, I favour Classic over Future. It would fulfil my personal Amiga needs. Also, while I'd sell an elderly relative to get my hands on a modern Amiga-like system, I don't have faith that it would happen. All new OSes, especially those of an OSS variety always seem to end up being just another unix-a-like. This has even happened to OS4 in certain places (thankfully outside the core system).
Back to Classic, I would propose taking a similar approach to the MOS team in getting the project going. There really needs to be momentum from the start. Start off as simply a bunch of replacement files that goes on top of an OS3.1 install and keep building them up until you have replaced everything. Make sure that this can be done easily and painlessly (see AIAB) and you're sorted. People want to use stuff now, not when it's done. Providing a fully function system from the start is key here. Really, it's how AROS should have been done from the start.
I guess that about covers it. Although it's great that MOS and AOS are still progressing, the whole community/market/whatever-you-wanna-call-it is far too fragile for anything other than an open source operating system at the heart of it. We really need AROS to do well.
Sorry for the extra-long post.
-
Isn't the only reason why AROS hasn't vanished into oblivion years ago the fact that it runs on x86 ? Other then that what makes it different from OS4 / MOS ?
-
With the current lack of manpower, formally forking AROS will just create 2 stagnant projects. Besides, AfAOS sort of accomplishes what your proposed Classic fork would do anyway.
I don't follow AROS very closely, so I don't know if there are any roadblocks to memory protection. The problem with implementing it on the other Amiga OSes has always been that everything would break. Given that there probably isn't any useful AROS software whose source code has been lost, I say, break it, then fix. Finish up that UAE integration to open up the Amiga's back catalogue, and there you go: a usable Amiga operating system on standard hardware.
-
Can I be blunt too?
Are you on the aros developers mailing list?
Have you committed code to the svn repository?
What can you do to help move it forward?
-
lacking the icon...
beating dead horse...
sigh...I have FUN with my old A2000 ... but NO future (don't want one..fun is enough)
Tom UK
PS: Mac Powerbook Pro...
-
@Lemmink
AROS is open source, that is what males it different
@Matt_H
Matt_H wrote:
With the current lack of manpower, formally forking AROS will just create 2 stagnant projects.
As I mentioned, my personal preference would be to concentrate solely on Classic. It makes the most sense. Amiga is predominately a retro-hobby these days. Targeting a VM as opposed to real hardware would solve many nightmares. UAE is a pretty decent bit of software ;-)
Besides, AfAOS sort of accomplishes what your proposed Classic fork would do anyway.
No it doesn't. AfAOS replaces a couple of bits and pieces, not the whole system. I mean absolutely no disrespect to Bernd and his hard work here, but it's also a bit of a bodge. A full enhanced 68k port would be much better.
I don't follow AROS very closely, so I don't know if there are any roadblocks to memory protection. The problem with implementing it on the other Amiga OSes has always been that everything would break. Given that there probably isn't any useful AROS software whose source code has been lost, I say, break it, then fix.
Which is a mammoth task. Everything would break. Whether or not you have the source doesn't matter all that much. The main thing is that currently the Devs won't break it because source compatibility is still core to the 'vision'.
And if they did break it what would happen to AfAOS? You can't have it both ways without a split.
Finish up that UAE integration to open up the Amiga's back catalogue, and there you go: a usable Amiga operating system on standard hardware.
As far as I can see UAE integration is a fluffy after-thought that has been touted to stop people complaining about the lack of Amiga software. Even if it were to happen you'd still need Classic to run any software on it. It would be pretty rubbish if an open source OS required a commercial OS to run the majority of it's software.
Going back to classic again, with a minimal host OS (KXLight / XAmiga) and UAE you could have a usable Amiga Operating system on standard, or any other hardware you wanted.
-
Interesting how the same thread can come out over and over again.
Six months ago, Rob, the responsible of the Traveller project wrote on its blog a similar proposal, despite based from different concepts: you can find it here (http://cataclysm.cx/2008/02/01/i-dont-want-your-money-i-just-want-your-love/), and here (http://aros-exec.org/modules/newbb/viewtopic.php?topic_id=2608&forum=4) you can find the subsequent discussion in the forum.
Inside the same thread a proposal from HenryCase in order to avoid a fork and provide both new ways to evolve AROS (such as the holy memory protection grail) and in the same time provide the compatibility with the 3.1 API, but cannot say more about it because he haven't laid out anywhere yet, well maybe this is the right time....
By the way, i guess the post of rob is quite nailing the point: the AROS users and developers bring the same split feelings that Amiga/morphos user bring, therefore what is stopping now AROS to be more developed is the lack of a common goal.
And the fact that some of the assigned bounties were dropped without being brought to completion and not even shared what done so far does not help (tigger and EvilRich around?).
I personally think that AROS should more focus on evolving the amiga os way than simply provide an environment where to play older stuff: it has the potential and should use it.
The problems in a fork are: first that AROS is still incomplete, despite progress is slowly made and, as stated above, the low activity of the community that will slow down even more the activity with the forked project.
But,probably, a fork will not be needed:the latest homebrew hardware events, such as Natami that will lay on Amiga OS but especially on AROS in order to use the superAGA fucntionalities, should provide the way to increase the activity in the integration of old apps with the new system and proceed on porting AROS even on the classic hardware: part of the work has been made with AfAOS, we still need the kickstart and hardware specific code libraries.
At the end what i hope will have, and honestly is what developers need, is a base API reference to write the applications that should be easily portable on all the amiga/morph/AROS platforms and therefore an increase of the available software library and the coming of new developers.
Saimon69
-
yakumo9275 wrote:
Can I be blunt too?
Of course you can.
Are you on the aros developers mailing list?
Not anymore.
Have you committed code to the svn repository?
If I think a project's overall direction is flawed, am I going to waste time submitting code that won't change that? Would that really be a good use of my time? Or are you suggesting that I do the whole lot myself?
I fail to see what that really has to do with it. I really wish that people would stop hiding behind open source as if it is some sort of licence to absolve their projects of any criticism. If this didn't happen in the first place I wouldn't have to be so blunt in my original post.
What can you do to help move it forward?
Proposing a sensible path forward (see above - you did actually read it all before going onto the defensive didn't you?)
-
uncharted wrote:
If I think a project's overall direction is flawed, am I going to waste time submitting code that won't change that? Would that really be a good use of my time? Or are you suggesting that I do the whole lot myself?
There is no direction! The only goal is to get something that works, and each dev works to improve what they can, what they know how to improve...
I appreciate you don't have much respect for the project, I'm quite sure it offers you nothing... but if you compile the latest sources, you end up with a usable OS, that is in every way like AmigaOS... with a bunch of programs to play with... frankly that's pretty much all my Amigas can do too... I wouldn't get much more from any other Amiga system...
I would like a 68k build, and there are people messing around with that right now... but no Amiga system can really offer any more than than what AROS offers now... so I am happy.
-
uncharted wrote:
I fail to see what that really has to do with it. I really wish that people would stop hiding behind open source as if it is some sort of licence to absolve their projects of any criticism. If this didn't happen in the first place I wouldn't have to be so blunt in my original post.
What it tells me is you have no investment in AROS, and it is some minor form of absolvment from criticism, because if you dont like it, you have all the rights in the world to go do something about it, make your own fork. Start writing code to get that fork where YOU want it to be.
Proposing a sensible path forward (see above - you did actually read it all before going onto the defensive didn't you?)
I did read it all, and I dont think I was defensive, I just asked you a couple of questions. I can understand your POV. You want it NOW, not when its ready. I want a lot of things now too, but the world doesn't work that way. We have to work hard for the things we want.
Proposing a 'sensible path forward' does what exactly? what does that achieve? nothing... in open source, code is what counts, being the guy with the big idea doesn't really mean squat.
AROS only has a small set of developers. If you fork it, how many folks do you think will work on the classic side? how many on the future side? Will it take twice as long to get anywhere with half the developers?
What would this 3.1classic compatible AROS bring that genuine 3.1 doesn't? You can still get 3.1 on amiga forever cd so its not like its unavailable.
Why dont you start an arch/m68k-amiga port in aros, because, that would get you off and going with the whole 'classic' piece that you want. Once you get that compiling you should be able to drop in all the libs and other bits as replacements.
I do like the idea of a 'future' version, Id love to see an L4 kernel driving it, make use of that mmu, a much better filesystem, etc.
I'm trying to get aros working on the efika. I dont care about classic hardware or classic binary compat. I would love to see a jittable emulation layer that translates api calls across from m68k apps into native code. But I dont really see your 'classic' view point of 100% drop in binary compat and replacing wb3.1. You really need to explain your classic idea and sell it coz I dont see the point of it. If genuine 3.1 wasnt available, then I could maybe see a point to it.
The way I see it, AROS now is the start of your 'future'. The 3.1API is a nice API to start with, its well defined and understood for app developers. I'm very much with Rob on his views.
-
Matt_H wrote:
With the current lack of manpower, formally forking AROS will just create 2 stagnant projects.
Bingo.
saimon69 wrote:
Inside the same thread a proposal from HenryCase in order to avoid a fork and provide both new ways to evolve AROS (such as the holy memory protection grail) and in the same time provide the compatibility with the 3.1 API, but cannot say more about it because he haven't laid out anywhere yet, well maybe this is the right time....
Well remembered. I'm not going to say any more on the subject until I'm ready to code a 'proof' for my theory (the MP issue always causes a sh*tstorm), so you're right, now is not the time.
uncharted wrote:
Proposing a sensible path forward (see above - you did actually read it all before going onto the defensive didn't you?)
uncharted, thank you for ideas, but as you can see from saimon69's post AROS fans already discussed these same ideas a few months ago. Some would argue we lost a good developer out of those discussions (though there were other factors at play too).
I'd say yakumo9275's response was justified. Too many armchair experts (including myself), not enough developers, that's AROS's main problem, so what are you going to do to fix that?
Anyway, contribute something concrete to AROS and I'm sure you'll get a more favourable response to your ideas.
-
yakumo9275 wrote:
You really need to explain your classic idea and sell it coz I dont see the point of it. If genuine 3.1 wasnt available, then I could maybe see a point to it.
I know you were talking to uncharted here, but what the hell, let's open this can of worms again...
68k AROS + new 68k Kickstart = freedom for classic machines, giving a much more expandable and customisable OS. Useful for the machines we owe so much to as well as the new 68k Amigas (Minimig, Natami, etc...). To turn your question around, why would you not want that?
-
My 2 bits . . .
Many years ago I asked why AROS isn't designed for one piece of hardware and complete the OS so it can be used regularly, then branch out to more hardware. But every one back then wanted it to run on their hardware, hence, still under development for all that hardware.
AROS for 68k, a great path to go for freedom from 3.x series with more modern possibilities is great. My preferred path for AROS ATM. Think Natami . . .
But going in so many directions just slows the development overall.
I do like tinkering with AROS but its not even going to make my 6 year plus predicted "2010 Ready For Prime Time" debut.
Nothing is really happening in the Amiga world ATM.
Yes, I realize MorphOS 2.0 has released, but where's the hardware. The price . . . :crazy: Efika . . . :lol:
OS4 is a deadstick ATM.
AROS has stalled.
Ho Hum . . . . . . . .
-
But going in so many directions just slows the development overall.
I think that is hitting the nail on the head; any development effort can be thwarted by trying to make the software too many things to too many people. It's important to focus on setting realistic milestones when implementing anything, an operating system especially. it seems to me that the efforts of the community are far too divided for anyone's good.
Yes, I realize MorphOS 2.0 has released, but where's the hardware. The price . . . :crazy: Efika . . . :lol:
don't laugh, the efika is a nice little system. I'm thinking about buying a second one, actually - my fiancee loves the idea of the low power consumption and only needs basic functions (word processor and email). the morphos team promised 2.0 the second quarter and delivered despite some, myself included, thinking they would not; I think I will not be as surprised if we see MOS on Mac Mini systems soon. :)
-
Wolfe wrote:
AROS for 68k, a great path to go for freedom from 3.x
...and no hardware to buy. Great idea. I think we already have that. :roll:
-
HenryCase wrote:
Matt_H wrote:
With the current lack of manpower, formally forking AROS will just create 2 stagnant projects.
Bingo.
Hahaha, but no... Go on fork it now, it's easy! Download the souce code, set up an SVN server, and then develop it how ever you like... Try and get devs on board though and see how far you get... If you make a sucess then what have we lost?
But like most in the Amiga scene it's all just talk... at least the AROS team as it is DOES achieve something.
I summise that the current AROS devs just work in their spare time and work on what parts they understand or find interesting. They do a fine job. If you make fork, they're not going to work on that fork (they don't have the time or the motivation)... the Main AROS tree wil develop as it always has, and your fork will die.
saimon69 wrote:
Inside the same thread a proposal from HenryCase in order to avoid a fork and provide both new ways to evolve AROS (such as the holy memory protection grail) and in the same time provide the compatibility with the 3.1 API, but cannot say more about it because he haven't laid out anywhere yet, well maybe this is the right time....
Well remembered. I'm not going to say any more on the subject until I'm ready to code a 'proof' for my theory (the MP issue always causes a sh*tstorm), so you're right, now is not the time.
It can't be done... the AmigaOS design does not allow for MP... I am happy to talk you through it again... it's a topic I actually enjoy talking about.
uncharted wrote:
Proposing a sensible path forward (see above - you did actually read it all before going onto the defensive didn't you?)
uncharted, thank you for ideas, but as you can see from saimon69's post AROS fans already discussed these same ideas a few months ago. Some would argue we lost a good developer out of those discussions (though there were other factors at play too).
Yes, we lost a great developer... but even though he has now left, he contributed a great deal to the AROS project!
I'd say yakumo9275's response was justified. Too many armchair experts (including myself), not enough developers, that's AROS's main problem, so what are you going to do to fix that?
I don't think AROS actually has a problem at all... we are half way throguh 2008... the whole concept of the Amiga is VERY VERY dead.
AROS is now very much a hobby project, and as such it's actually doing very well. I can install it on my various computers, it functions just as my old Amigas function, it's fun to play with... it costs me nothing to mess around with... I have the source code if I want to try out an operating system idea, I can try it out, because of AROS I understand how AmigaOS actually works and the design choices made and why... and seeing the flaws in it made me apprciate the design choices and trade offs made in other operating systems...
Anyway, contribute something concrete to AROS and I'm sure you'll get a more favourable response to your ideas.
Anyone is free to contribute to AROS, and also use AROS as a learning tool!
-
Colani1200 wrote:
Wolfe wrote:
AROS for 68k, a great path to go for freedom from 3.x
...and no hardware to buy. Great idea. I think we already have that. :roll:
My thought on this was for the future . . .
I have an A1200 for it until Natami :-D Arrives . . .
-
pkillo wrote:
Yes, I realize MorphOS 2.0 has released, but where's the hardware. The price . . . :crazy: Efika . . . :lol:
don't laugh, the efika is a nice little system. I'm thinking about buying a second one, actually - my fiancee loves the idea of the low power consumption and only needs basic functions (word processor and email). the morphos team promised 2.0 the second quarter and delivered despite some, myself included, thinking they would not; I think I will not be as surprised if we see MOS on Mac Mini systems soon. :)
But its not fast enough to replace my A1200 and all its games etc . . . My migi is getting old. Now if all I needed was to check mail and surf the web etc., well, still, the cost of the OS :crazy: Just buy a chesp PC and run BeOS . . . :-D Or Zeta like me . . . As I don't like Linux . . :-x
Now, when they release a version for the Mac Mini 1.5 (which I have and use side by side with my Zeta Box) I might buy it since I already have the hardware and can get more use out of Morph OS . . . :-)
-
@bloodline
bloodline wrote:
Hahaha, but no... Go on fork it now, it's easy! Download the souce code, set up an SVN server, and then develop it how ever you like... Try and get devs on board though and see how far you get... If you make a sucess then what have we lost?
I agree that creating a fork isn't necessarily hard (or a bad thing for that matter) but I would prefer to keep the number of forks to a minimum. Where do you think I would be getting those developers from, the AROS dev community or elsewhere? If I'm going to get new devs involved, I'd much rather they worked on the main x86 (or x86-64) AROS branch.
bloodline wrote:
But like most in the Amiga scene it's all just talk... at least the AROS team as it is DOES achieve something.
I agree, and that's why I admire the AROS devs, they're making a real positive difference in the Amiga community.
bloodline wrote:
I summise that the current AROS devs just work in their spare time and work on what parts they understand or find interesting. They do a fine job. If you make fork, they're not going to work on that fork (they don't have the time or the motivation)... the Main AROS tree wil develop as it always has, and your fork will die.
The only (new) fork I would be interested in would be a native 68k port. There are other projects I would rather complete first as I don't possess all the necessary skills to work on a 68k fork. The first major project I'd like to tackle is an AROS audio player, though I must admit that I'm still in the early stages of learning C programming properly.
bloodline wrote:
It can't be done... the AmigaOS design does not allow for MP... I am happy to talk you through it again... it's a topic I actually enjoy talking about.
Glad you feel that way bloodline, I didn't want to feel like there was a negative vibe after our last discussions on the subject (and other subjects), but I did feel that vibe, which is one of the reasons I held off.
So lets start this discussion again, but first a question... can you name me two Amiga programs that are designed to trade data with each other through memory addresses?
bloodline wrote:
Yes, we lost a great developer... but even though he has now left, he contributed a great deal to the AROS project!
Yes, Robert N did contribute a great deal to AROS, but to those who are unaware of the situation it's best we clarify the outcome of the 'AROS fork' discussion he started was not the main reason he left us.
bloodline wrote:
I don't think AROS actually has a problem at all... we are half way throguh 2008... the whole concept of the Amiga is VERY VERY dead.
AROS is now very much a hobby project, and as such it's actually doing very well. I can install it on my various computers, it functions just as my old Amigas function, it's fun to play with... it costs me nothing to mess around with... I have the source code if I want to try out an operating system idea, I can try it out, because of AROS I understand how AmigaOS actually works and the design choices made and why... and seeing the flaws in it made me apprciate the design choices and trade offs made in other operating systems...
AROS doesn't have a problem in the sense that it is still alive and progressing, but it has great potential that is still untapped, which we would be able to tap with more developers. In my opinion AROS should be the most popular of the three modern Amiga based OS's, why do you think that it is not?
-
I'll throw my 2 cents in here...
AROS is absolutely usable, looks great, performs well, etc. It has for a long time now. That said I think all development needs to cease, and be re-focused on 3 vital projects:
Modern web browser (started, but needs more hands)
JAVA runtime environment (I see no mention of this anywhere)
Multi-IM client (at least Yahoo and G-Talk)
Take care of these projects and everything else will fall into place quickly. Give us web access and chat capabilities. Getting JRE working we'd instantly have a lot more applications to run.
This gives us BASIC usability using the old PCs most of us already have. With that, more programs will be written as more people will be interested in it, some of them programmers.
Look at BeOS, how much stuff was written for that before it tanked? Just because people were using it. QNX, looked promising till the morons who owned it closed it off. Linux's problem is it's too complex for the average user's desktop... Yet look how much development goes into programs for Linux, just because it CAN be used.
AROS gives us the ease of AmigaOS, with the power and cost effectiveness of x86. We don't need it on dead architectures, we need it on x86. We don't need backwards compatibility, we need to move forward. We don't need to run Amiga programs on it, we have UAE and real Amigas for that.
Give me my AROS!
-
Wolfe wrote:
But its not fast enough to replace my A1200 and all its games etc . . . My migi is getting old. Now if all I needed was to check mail and surf the web etc., well, still, the cost of the OS :crazy: Just buy a chesp PC and run BeOS . . . :-D Or Zeta like me . . . As I don't like Linux . . :-x
Do you mean it's not fast enough to emulate an A1200? Because it's certainly faster than one. I don't understand what you are saying - it's a PPC chip at 400MHz, which is much faster than an 68020, and still faster than any accelerator you could put into a 1200. Please explain.
As far as the cost of the OS, well, the way I'm figuring it, it's cheaper than Linux. An EFIKA costs $4.38 to run for a year, assuming it's on 24 hours a day, 365 days a year. A PC will cost (worst case) 100x that to run, or $438. (My PC has a 500 watt power supply, the EFIKA a 5 watt power supply. Many people now have PSUs in their PCs that draw even more power than that.) Even if my PC only consumes half that power (unlikely given it's almost never idle) I'm still looking at $219 in the first year of ownership. So, after a year, I've broken even buying MorphOS for an EFIKA, assuming it takes the place of a PC running Linux. If it runs Windows instead, then the PC could have a 300 watt PSU and the numbers come out the same, as Windows costs about $90.
btw, I'm curious about this Zeta os you mention. I did a quick google and didn't find anything. AFAIK BeOS lacks support for modern hardware and Haiku is not ready yet. I would like a nice alternative to Linux - I don't hate it like you say you do, but I do not believe it belongs on the desktop of anyone but a hacker. :)
Now, when they release a version for the Mac Mini 1.5 (which I have and use side by side with my Zeta Box) I might buy it since I already have the hardware and can get more use out of Morph OS . . . :-)
Well, I don't think you'll have long to wait. We should see it the fourth quarter of this year, hopefully. :) I too have a Mac Mini (1.42 though) that I'd like to put back in service with a new OS.
-
the problem with mac minis for morphos if they do it, 1 - the stock mini has barely any video ram onboard (some models 32mb some 64mb), and upgrading that is a nightmare,sice you cant just stick in a pc agp card, you need one with a mac bios (afaik...). different cards manufacuters use different ram chips so a mac compat bios is not often easy to get etc.
-
@yakumo: it's true, this will be a limitation, but my 1.42 mini was able to play back dvds and avi and mpeg files without problem while running all the extra overhead of OS X, so I think it would perform nicely enough running MorphOS. and, on a subjective level, when I compared the video performance of my mini (when i first got it) to a PC with the same radeon 9200 but with 128mb of vram the Mac was holding its own. couldn't tell you why, it just did.
for gaming, maybe this would be a larger issue but I don't see gaming being the application of choice on MorphOS. If you want to game, you're going to run out and buy a high-end PC so you can run your favorite MMORPG or multi-player shoot-em-up or else you're going to use a console system like the Wii or PS3.
-
@bloodline
Don't worry about answering my previous question, let's start the MP discussion now.
Okay, I'd like to start this post by making a couple of things clear. First of all, there are three separate questions we could discuss (as far as I can see) when it comes to the possibility of MP:
1. Can it be done?
2. Can it be done elegantly?
3. Should it be done?
I hope you will agree that there is little point answering questions 2 and 3 if we do not agree that the answer to question 1 is yes, so that's what we should focus our discussion on IMO (at least at first). What this means is that if the MP implementation is a complete bodge job but works successfully then it can satisfy the criteria for answering question 1 and we can move on to the 'should we/how best do we' do it part of the discussion.
Next, I'd like to outline what I'd class as successful MP. To me the job of MP isn't to make individual applications less prone to crashing, but rather to ensure that when applications crash they don't take down the operating system and other applications with them. That is the base functionality of MP IMO, do you agree? I would also say that close to that core MP functionality is the ability for the OS to attempt to recover the data the application was using before the crash, but that is not a function I will focus on for now.
==============================================================
Right, down to business. A program in AROS is loaded. The memory manager assigns a unique ID to that program. The program starts requesting RAM space from AROS to store its data. These calls will either be for:
A space in memory of a specific size but undefined location.
or...
A space in memory of a specific size and location.
The AROS memory manager I am proposing doesn't give programs a memory address based on the location requested, but rather provides a 'virtualization' layer. When the program then tries to call data from a memory address it thinks is set in RAM, the memory manager calls the data from where it is really stored and presents it to the program. The program doesn't know the difference.
Lets take an example to illustrate this further. The program Lunapaint is loaded. AROS assigns Lunapaint the ID 0111. Lunapaint then requests a 5-bit memory address space starting at address 3FDF. That address is already taken in RAM by the program Wookiechat, but this doesn't matter to us. The 5-bit memory space is put in the address starting A21C instead. When Lunapaint tries to read or write to the memory space it thought was at 3FDF, the memory manager knows that the data stored in 3FDF for program ID 0111 is actually at A21C and pulls this data instead.
So what does this mean? It means the OS has taken control of memory allocation away from programs without letting the programs know there is anything different happening. Now lets look at what happens when programs try to pass data between each other through memory addresses.
What we want to be able to define here are the processes that should be allowed to share memory addresses with each other, and those programs that should not be allowed to share memory addresses (prevent problems like buffer overflows for example). The approach I'm proposing here is what I'd call the 'AI' approach, even though the intelligence is not artificial... I propose we use our own brains!
When two programs try to write to the same address space, we get a dialog box pop up on AROS (much like an Internet firewall, such as ZoneAlarm) asking whether we would like for these programs to be able to share memory address space. Taking the Lunapaint/Wookiechat example used above, the dialog box would ask us if Lunapaint and Wookiechat should share address spaces. Our options could be something like:
Share 3FDF, no universal share.
Do not allow programs to share any memory address spaces.
Allow both programs full access to each others memory address spaces.
You might think this would be a pain to set up (having a dialog box pop up every time you open two certain programs at the same time), but a way around this is to use patches. How would this work? Well for patches to be applied the memory structure needs to be consistent every time the program is run (so that if I apply a patch setting to 3FDF one day the patch will work on the same data the next day). However, as we have 'virtualized' memory allocation this is possible. The memory manager is the OS layer doing all of the assigning of memory locations. When a program asks for a memory address in a specific location, we give it a memory location with that label. When a program asks for memory but doesn't specify where that memory should be we make sure that the 'virtualized' memory allocation follows rules, so that we will always be able to find that memory.
For example, on program boot up we are dealing with requests for memory that are always in the same order, so why not give them the labels 0001, 0002, 0003, 0004, etc... From this point on the memory needs of the application are dynamic, so we look at assigning memory labels not on sequence but on a mathematical algorithm based on how they are first assigned and the memory spaces already in use (there may be an easier way of doing this, basically I'm looking for ways to pinpoint the setting of a variable in a program).
Assuming the above variable pinpointing is possible, then we can apply our patches and they can work consistently every time. The memory labels assigned don't affect real memory usage in the sense that all programs can have the 0001, 0002, 0003, etc... labels and the real memory can still be fully utilised.
What would we use these patches for? Well, let's say we never want Lunapaint and Wookiechat to ever share memory addresses. We can create a patch for this. We can also create a patch so that Lunapaint will share certain memory addresses with Wookiechat but won't share those same memory addresses with UAE. The beauty of the patch system is you can make your system as secure (or as open) as you like. You could also distribute these patches between AROS users (like the way WHDLoad patches are created and distributed).
Okay, so we've looked at taking control of memory allocation, and how we can let programs interoperate in memory with our permission, the final piece of the puzzle is how we handle application crashes. Let's say I write a program for AROS, but I make a mistake so that in certain circumstances I'm trying to write a long string into a space defined for an integer, which causes the program to freeze/crash due to a buffer overflow.
Where does this buffer overflow go? How do we know the program is misbehaving? We have options here. We could let the program write the extra bits into a new 'virtual' address, hoping that the program will run smoothly afterwards (and have rules about how often this 'overflow' storage can happen before we flag up an error to the user) or we could simply flag an error as soon as the the buffer overflow happened. Link this with CPU throttling (so that programs caught in an infinite loop can have their CPU time reduced to avoid crashes) and you have a system that allows you to stop any misbehaving programs.
==============================================================
So we've looked at memory allocation control, memory permissions and crash control. Is there any other factors an OS needs for MP?
Note that the only part in my mind with a question mark over it is how to flag individual variables for 'patching', I'll have to look at how other systems (like compilers) do this, but even if this is not possible you'd still be able to have interoperability settings at the application level (i.e. Program 1 can share memory addresses with Program 2 but not with Program 3), which is still useful for system stability.
Please let me know what you think bloodline. Thanks.
-
At some point, most people eventually learn when the fundamental architecture of a system is no longer worth saving...
-
koaftder wrote:
At some point, most people eventually learn when the fundamental architecture of a system is no longer worth saving...
That falls into the 'should it be done' MP question category. Do you have any input on the 'can it be done' MP question (based on my last post in this thread)?
-
@yakumo9275
the stock mini has barely any video ram onboard (some models 32mb some 64mb), and upgrading that is a nightmare,sice you cant just stick in a pc agp card, you need one with a mac bios (afaik...). different cards manufacuters use different ram chips so a mac compat bios is not often easy to get etc.
Excuse me but where exactly are you going to plug this AGP card?
-
Piru wrote:
Excuse me but where exactly are you going to plug this AGP card?
dang, it all says it has a 4x compatible agp... but I guess its still inbuilt :( mybad. i'll leave this post for posterity :) i see it inbuilt on the underside of the motherboard
-
pkillo wrote:
Do you mean it's not fast enough to emulate an A1200? Because it's certainly faster than one. I don't understand what you are saying - it's a PPC chip at 400MHz, which is much faster than an 68020, and still faster than any accelerator you could put into a 1200. Please explain.
Amiga software that does hardware banging (emulating the real thing) would be toooooooo slow.
As to energy savings, thats cool, but the OS is tied to the hardware, so, you will be buying anither copy for the Mac?
btw, I'm curious about this Zeta os you mention. I did a quick google and didn't find anything. AFAIK BeOS lacks support for modern hardware and Haiku is not ready yet. I would like a nice alternative to Linux - I don't hate it like you say you do, but I do not believe it belongs on the desktop of anyone but a hacker. :)
Search BeOS Zeta in Google. It is BeOS enhanced (some anyway) but it went under due to legal issues. On the cheap was cheap hardware with BeOS - Modern hardware is more expensive. I have a Dell 2350 P4 (got it free - recycled), added max memory (1GB) and a PCI NVidia Graphics card (64 MB) All new for under $50. Add in the cost of the OS, and I still haven't spent much more than an Efika Mobo, without a case, PSU, video etc . . . . And BeOS Max is on another partition, as well as Amiga Forever 08, and room for more . . .
Well, I don't think you'll have long to wait. We should see it the fourth quarter of this year, hopefully. :) I too have a Mac Mini (1.42 though) that I'd like to put back in service with a new OS.
Lets hope so . . . :-D I still use my mini daily with OS X . . . With that Mini there is no need to buy an Efika for e-mail etc . . . :-) MorphOS is nice though . . :-)
-
Wolfe wrote:
Amiga software that does hardware banging (emulating the real thing) would be toooooooo slow.
Fair enough. But how much non-game software fits that bill? The people who want green computers generally aren't gamers. :)
As to energy savings, thats cool, but the OS is tied to the hardware, so, you will be buying anither copy for the Mac?
Absolutely. The mini is a nice rig, it just needs a less brain-dead approach to its operating system.
Search BeOS Zeta in Google. It is BeOS enhanced (some anyway) but it went under due to legal issues. On the cheap was cheap hardware with BeOS - Modern hardware is more expensive. I have a Dell 2350 P4 (got it free - recycled), added max memory (1GB) and a PCI NVidia Graphics card (64 MB) All new for under $50. Add in the cost of the OS, and I still haven't spent much more than an Efika Mobo, without a case, PSU, video etc . . . . And BeOS Max is on another partition, as well as Amiga Forever 08, and room for more . . .
Cool; I checked out a couple of pages on it. Looks fun; I'll have to check the hardware compatibility list and see if I own anything it'll run on!
-
Regarding memory protection:
1. Can it be done?
2. Can it be done elegantly?
3. Should it be done?
1. It can only be done part way and it requires an MMU.
2. No, managed code is much more elegant even if it is sometimes a bit slow.
3. No. If we are going to support the Natami and MiniMig as a starting place, we're going to have to realize that the fact that the 68070 on the production Natami and 68000 on the MiniMig will not have an MMU is going to prevent a memory-protected AROS from running at all!
There are so many things that should be done first:
1. Standard Template Library for C++ code. This can be implemented using STLPort.
2. Expand the runtime library to do STL functions in a memory-efficient manner. This can be implemented through template specialization.
3. Minimal bytecode as a distribution medium for code across all of the supported hardware platforms. (Possibly even portable to all of the OSs that support AmigaOS 3.1 APIs.) Tough one but but I favor LLVM to implement this.
4. Compiler to generate code in that bytecode. LLVM-GCC does this but is difficult to build. The upcoming Clang project on the Mac might someday be a better choice though.
5. Entry level programming languages. Another tough one but I'm already working on this one. Hollywood also works for this one.
-
SamuraiCrow wrote:
Regarding memory protection:
1. Can it be done?
2. Can it be done elegantly?
3. Should it be done?
1. It can only be done part way and it requires an MMU.
2. No, managed code is much more elegant even if it is sometimes a bit slow.
3. No. If we are going to support the Natami and MiniMig as a starting place, we're going to have to realize that the fact that the 68070 on the production Natami and 68000 on the MiniMig will not have an MMU is going to prevent a memory-protected AROS from running at all!
SamuraiCrow thanks for your reply, but you're jumping ahead in the discussions here. As I said before I see no point discussing questions 2 and 3 until we come to an agreement on the answer to question 1. The only way you're going to convince me that MP can only be done partially is to find flaws in the MP solution I outlined earlier in this thread. Please do so. Thanks in advance.
-
@HenryCase
One problem in memory protection is that applications and the operating system are sharing system structures. For example when you call OpenWindow() the system returns struct Window which is globally shared. If your application trashes this structure it may lead to crash of operating system.
The design of Amiga is unfortunately flawed for memory protection. One could say that on Amiga there is no user space and kernel space -- there is only kernel space where all applications are running.
-
Oh no! Another "how should other people work" topic about AROS.
And another fork/split proposal. It seems the Amiga community can't live without splitting things in thousands pieces.
Well, for people that aren't aware of it, AROS has its own mission and definition, which is being "a lightweight, efficient and flexible desktop operating system, designed to help you make the most of your computer. It's an independent, portable and free project, aiming at being compatible with AmigaOS 3.1 at the API level (like Wine, unlike UAE), while improving on it in many areas".
Isn't it enough? Well, there are the ending lines too, "The source code is available under an open source license, which allows anyone to freely improve upon it" which formally and actually allow anyone willing to do his/her "better AROS" to take the code, modify it and reach the goal he/she likes.
So, having clear in mind that all in all AROS is a *hobbystic* project, the community can do different things with it:
1. just whining and claiming their opinion would be the best way to improve AROS
2. contribute with code to reach the common goal stated above.
3. take the existing code and improve upon it the way they like, just to become the new ruler of the world.
The direction is already here for the people that want to see it. The others may disagree, but please don't claim AROS is pointless or directionless anymore, as it is completely untrue.
-
itix wrote:
@HenryCase
One problem in memory protection is that applications and the operating system are sharing system structures. For example when you call OpenWindow() the system returns struct Window which is globally shared. If your application trashes this structure it may lead to crash of operating system.
Thanks for your reply itix. I looked into the OpenWindow() API call and I can't see anything that would prevent changing it from a global variable to a more local one.
itix wrote:
The design of Amiga is unfortunately flawed for memory protection. One could say that on Amiga there is no user space and kernel space -- there is only kernel space where all applications are running.
This is the case now, but we're looking at ways to create a user space-kernel space separation whilst keeping OS3.1 API compatibility. This is not your daddy's exec if you catch my drift. If it is possible to implement a new kernel with MP that supports the OS3.1 API calls then we can have MP in an Amiga based OS.
So a good question is this: are there any global variables that cannot be contained more locally?
-
paolone wrote:
The direction is already here for the people that want to see it. The others may disagree, but please don't claim AROS is pointless or directionless anymore, as it is completely untrue.
Paolone, I don't know if I was meant to be one of the targets of your 'all talk no action' rant or not, but I wanted to tell you that I really believe VmwAROS is a great asset to the AROS (and Amiga) community, and is helping to bring some structure back to AROS application development. Thank you for your continued work on it.
-
@HenryCase
Thanks for your reply itix. I looked into the OpenWindow() API call and I can't see anything that would prevent changing it from a global variable to a more local one.
Local in which way? Intuition reads and writes to this structure and application reads and writes to this structure. There is no way you could make it "local" and retain API compatibility.
If it is possible to implement a new kernel with MP that supports the OS3.1 API calls then we can have MP in an Amiga based OS.
If it was possible AROS would have implemented it already.
-
Henry...
There is only one small conceptual point that you are missing. I have been racking my brains to try and find a good metaphor for the problem... bur I can't really find one.
I want to address your conceptual error as you raised it before... That is this strange idea of your that a program requests data from memory. It doesn't.
What we have is a big flat area of memory, and a CPU.
The CPU does not know or care where it is reading and writing, it simply follows what it is told to do by a computer program. The CPU in AmigaOS has no information regarding the status of the code at all. While AmigaOS offers contexts (where each exec task has it's own set of certain data structures etc), these contexts are all part of the same address space.
In the AmigaOS design, the designers decided that they wanted the system to be fast and use as little memory as possible. Thus AmigaOS was designed with the idea that any data structure that could be shared, would be shared... No provision was made in the design of Amiga to control the access to system wide data structures... this was a design decision, and it can't be retro fitted without totally redesigning the system, and it would no longer be AmigaOS compatible.
You are thinking high level, not machine level... Your ideas are actually an area of research called "managed code", the most famous is MicroSoft's Singularity project. There programs access memory via defined system functions thus memory access can be controlled without the need for an MMU.
But AmigaOS is not a managed code system, it was designed as an operating system to be programmed at a machine code level... in fact I can write a program in ASM that would work on AmigaOS in a matter of minutes... I certainly couldn't even begin to think about doing that on a real operating like a UNIX or Windows... where I would have to stick with C.
-
itix wrote:
Local in which way? Intuition reads and writes to this structure and application reads and writes to this structure. There is no way you could make it "local" and retain API compatibility.
Can you talk me through the way OpenWindow() is used in OS 3.1 so that I can look at creating this separation based on how it actually works and not my basic interpretation of Amiga API guides from the Internet.
itix wrote:
If it is possible to implement a new kernel with MP that supports the OS3.1 API calls then we can have MP in an Amiga based OS.
If it was possible AROS would have implemented it already.
I don't believe that MP was/is inevitable. There were more important tasks to complete when the AROS kernel was designed and created, like just getting it to work.
-
HenryCase wrote:
itix wrote:
@HenryCase
One problem in memory protection is that applications and the operating system are sharing system structures. For example when you call OpenWindow() the system returns struct Window which is globally shared. If your application trashes this structure it may lead to crash of operating system.
Thanks for your reply itix. I looked into the OpenWindow() API call and I can't see anything that would prevent changing it from a global variable to a more local one.
But the data structures are shared... Every program ever written for AmigaOS and the System itself expect to have access to these data structures... what do you want to do? Give each one it's own local copy of intuition? How the hell could that work? The way intuition was designed doesn't allow for that!
itix wrote:
The design of Amiga is unfortunately flawed for memory protection. One could say that on Amiga there is no user space and kernel space -- there is only kernel space where all applications are running.
This is the case now, but we're looking at ways to create a user space-kernel space separation whilst keeping OS3.1 API compatibility.
There is no separation in the design of the API... the actual concept of the OS is everything running in Kernel Space... This isn't a push a button and the Kernel is in a separate space problem... there is no separation in the API design!
This is not your daddy's exec if you catch my drift.
No, you really need to spell this out because it is currently a hidden component in your theory and possibly one that is causing your confusion.
If it is possible to implement a new kernel with MP that supports the OS3.1 API calls then we can have MP in an Amiga based OS.
But those very same API calls return pointers to shared data structures... how do you get around that?
So a good question is this: are there any global variables that cannot be contained more locally?
That's an irrelevant question, you are thinking as if the CPU is running high level code, it is not.
-
Money, lots and lots of money spread over many years. If you can hire devs and guide them with vision of a great future, you can move AROS forward.
Dammy
-
@HenryCase
Any time you see a structure being passed into a system function with an &structurename in C code, it is passing a pointer to an internal stucture in the application's memory to the operating system. More importantly, if the operating system allocated the memory and passed back the pointer, the operating system would be the owner of that memory and would have to "change ownership" to the application and "change ownership" back to the operating system every time a pointer was passed back to it. That would be a glacially slow operation.
I think the answer to the issues that are often associated with memory protection are generally better addressed by using "safe" programming languages like Oberon and some versions of Basic that either don't directly support pointers in general, or have tight limitations on the use of arrays and other data structures to monitor them for overruns and NULL pointer values internally.
-
@SamuraiCrow
SamuraiCrow wrote:
@HenryCase
Any time you see a structure being passed into a system function with an &structurename in C code, it is passing a pointer to an internal stucture in the application's memory to the operating system. More importantly, if the operating system allocated the memory and passed back the pointer, the operating system would be the owner of that memory and would have to "change ownership" to the application and "change ownership" back to the operating system every time a pointer was passed back to it. That would be a glacially slow operation.
Anything running in supervisor mode should have full access to all memory in the system. Thus, the OS kernel wouldn't have to change ownership back and forth. For non-kernel OS modules that run in user mode, they could run directly in the application's task/process; this means that there still wouldn't be any ownership change.
I think the answer to the issues that are often associated with memory protection are generally better addressed by using "safe" programming languages like Oberon and some versions of Basic that either don't directly support pointers in general, or have tight limitations on the use of arrays and other data structures to monitor them for overruns and NULL pointer values internally.
The problem with that is that it requires all software to play by the rules. Malicious code does not. Memory protection helps prevent malicious code from doing harm by restricting access to resources.
Hans
-
@bloodline
Thanks for your response.
bloodline wrote:
What we have is a big flat area of memory, and a CPU.
The CPU does not know or care where it is reading and writing, it simply follows what it is told to do by a computer program. The CPU in AmigaOS has no information regarding the status of the code at all. While AmigaOS offers contexts (where each exec task has it's own set of certain data structures etc), these contexts are all part of the same address space.
I agree, the CPU doesn't have to care where it is reading or writing, it just has to follows what it is told to do by a computer program, namely the new AROS memory manager I am proposing. AROS is software too. I'm taking control over memory from the applications and giving it to the OS. If a program uses an API call like AllocMem() then AROS knows about it, right?
bloodline wrote:
In the AmigaOS design, the designers decided that they wanted the system to be fast and use as little memory as possible. Thus AmigaOS was designed with the idea that any data structure that could be shared, would be shared... No provision was made in the design of Amiga to control the access to system wide data structures... this was a design decision, and it can't be retro fitted without totally redesigning the system, and it would no longer be AmigaOS compatible.
Clearly I'm not aiming to clone every design decision made by the AmigaOS designers. I'm trying to focus on how we keeping OS 3.1 API compatibility in a MP system, we can discuss if the resultant OS is Amiga-like later.
bloodline wrote:
You are thinking high level, not machine level... Your ideas are actually an area of research called "managed code", the most famous is MicroSoft's Singularity project. There programs access memory via defined system functions thus memory access can be controlled without the need for an MMU.
Interesting. Can you give me an overview of how Microsoft's Singularity achieves this feat?
bloodline wrote:
But AmigaOS is not a managed code system, it was designed as an operating system to be programmed at a machine code level... in fact I can write a program in ASM that would work on AmigaOS in a matter of minutes... I certainly couldn't even begin to think about doing that on a real operating like a UNIX or Windows... where I would have to stick with C.
AmigaOS may not have been a managed code system historically, but we're not looking at what it was but rather what we can make it without breaking compatibility. You'd still be able to code in ASM with my MP system. The programs making AmigaOS API calls see the 'virtual' memory layout, but the users of the OS can see the 'virtual' memory layout and the 'real' memory layout. I hope you can see that gives us options to allow ASM code to run alongside other code.
One option is to add a new API call(s) especially for ASM code. When the ASM program is executed it first makes an API call that says something like 'I will need memory from XXXX to XXXX to run'. If this memory is free then it is assigned to the ASM program. If it is not free you can either:
1. Close the programs using that memory.
2. Move the programs using the memory to another memory space (the programs are paused during this move, but can carry on afterwards as normal due to the 'virtual' memory addresses not moving.
3. A copy of the ASM program is made, and this copy is amended with available memory address locations before being run (the application doing this works like a find and replace function in a word processor).
However, ASM by its nature does not allow MP(AFAIK), unless you sandbox it, so yes you could crash AmigaOS with a badly coded ASM app, but the same is true of any operating system. If you code ASM apps so that they stay out of the way of the area where the OS is running from you should be okay.
-
@bloodline
I'll reply to your last post shortly, outlining my ideas for the new exec will take a little time.
@Hans
Hans_ wrote:
Anything running in supervisor mode should have full access to all memory in the system. Thus, the OS kernel wouldn't have to change ownership back and forth. For non-kernel OS modules that run in user mode, they could run directly in the application's task/process; this means that there still wouldn't be any ownership change.
Correct. The memory in use would be jointly owned by the kernel and the applications permitted to run in that space. This allocation only changes when you close an application or give a new application rights to use that space.
-
HenryCase wrote:
@bloodline
Thanks for your response.
bloodline wrote:
What we have is a big flat area of memory, and a CPU.
The CPU does not know or care where it is reading and writing, it simply follows what it is told to do by a computer program. The CPU in AmigaOS has no information regarding the status of the code at all. While AmigaOS offers contexts (where each exec task has it's own set of certain data structures etc), these contexts are all part of the same address space.
I agree, the CPU doesn't have to care where it is reading or writing, it just has to follows what it is told to do by a computer program, namely the new AROS memory manager I am proposing. AROS is software too. I'm taking control over memory from the applications and giving it to the OS. If a program uses an API call like AllocMem() then AROS knows about it, right?
But in AmigaOS there is no way to control what a program does with that memory... It could quite happily pass the pointer returned by AllocMem() to another program... and that is a system legal thing to do in AmigaOS, this is a fundamental concept in AmigaOS that allow the system to function... if you have memory protection the pointer that the program received from the AllocMem() function doesn't exist anywhere but in that program's address space... so as soon as it tries to pass that pointer to another program/system function/etc it has just passed bad Data!! There is NO way in AmigaOS to know what a program is passing to another program.
bloodline wrote:
In the AmigaOS design, the designers decided that they wanted the system to be fast and use as little memory as possible. Thus AmigaOS was designed with the idea that any data structure that could be shared, would be shared... No provision was made in the design of Amiga to control the access to system wide data structures... this was a design decision, and it can't be retro fitted without totally redesigning the system, and it would no longer be AmigaOS compatible.
Clearly I'm not aiming to clone every design decision made by the AmigaOS designers.
Then you won't have AmigaOS... it would be like adding Gendered and Cased nouns to English... no existing English speaker would be able to use it... as it wouldn't be English anymore.
I'm trying to focus on how we keeping OS 3.1 API compatibility in a MP system, we can discuss if the resultant OS is Amiga-like later.
Amiga-like = AmigaOS 3.1 API compatibility. By the AROS definition.
bloodline wrote:
You are thinking high level, not machine level... Your ideas are actually an area of research called "managed code", the most famous is MicroSoft's Singularity project. There programs access memory via defined system functions thus memory access can be controlled without the need for an MMU.
Interesting. Can you give me an overview of how Microsoft's Singularity achieves this feat?
I already did... The concept of the memory pointer doesn't exist in a managed code system. All memory access is mediated by the Operating system. it's a slow way to operate, but it does allow stability on non MMU systems. The overhead of such a system make it unsuitable for real use... thus it was just a research project, now abandoned by MicroSoft, and released to the Open Source community.
Google and wikipedia are your friends here... I don't have to hold you hand the whole way?
bloodline wrote:
But AmigaOS is not a managed code system, it was designed as an operating system to be programmed at a machine code level... in fact I can write a program in ASM that would work on AmigaOS in a matter of minutes... I certainly couldn't even begin to think about doing that on a real operating like a UNIX or Windows... where I would have to stick with C.
AmigaOS may not have been a managed code system historically, but we're not looking at what it was but rather what we can make it without breaking compatibility.
what you are suggesting is the same as me saying... I shall try and cook a Chicken in a cement mixer... Just not possible... you would end up with mashed raw chicken...
You'd still be able to code in ASM with my MP system. The programs making AmigaOS API calls see the 'virtual' memory layout, but the users of the OS can see the 'virtual' memory layout and the 'real' memory layout. I hope you can see that gives us options to allow ASM code to run alongside other code.
Oops, I've confused you with my ASM example... I was simply trying to show how Simple the design of AmigaOS was... not that ASM was good or bad... One can easily write an computer program on Any OS in ASM (except a managed code system, were everything must be high level), but it's harder work because they are more complex.
One option is to add a new API call(s) especially for ASM code.
And what program is going to use a function call it doesn't know about!?!?
When the ASM program is executed it first makes an API call that says something like 'I will need memory from XXXX to XXXX to run'. If this memory is free then it is assigned to the ASM program. If it is not free you can either:
What? Why? This really makes no sense at all?!?
1. Close the programs using that memory.
??? Why... What if that program was your Email client?
2. Move the programs using the memory to another memory space (the programs are paused during this move, but can carry on afterwards as normal due to the 'virtual' memory addresses not moving.
How the hell do you move it? remember AmigaOS has no idea what program is using what memory...
You really must read about how virtual memory works as, this statement shows that you don't understand it at all!
3. A copy of the ASM program is made, and this copy is amended with available memory address locations before being run (the application doing this works like a find and replace function in a word processor).
Again, you haven't made any sense here...
However, ASM by its nature does not allow MP(AFAIK),
ASM has nothing to do with MP... that is a design consideration of the OS, not the programing language you have used.
unless you sandbox it, so yes you could crash AmigaOS with a badly coded ASM app, but the same is true of any operating system.
No it isn't... In a real OS... like OSX, no program can take down the OS... no matter how you wrote it. that is the point of MP.
If you code ASM apps so that they stay out of the way of the area where the OS is running from you should be okay.
That is the point of MP... so if a program tries to hit the OS space... it can't, this is done by sitting a program in it's own address space, where the OS does not exist... the App has no ability to see any other memory than what is in it's own address space... once the CPU is in that task's context it cannot access (read or write) outside of that context's address space.
-
HenryCase wrote:
@bloodline
I'll reply to your last post shortly, outlining my ideas for the new exec will take a little time.
@Hans
Hans_ wrote:
Anything running in supervisor mode should have full access to all memory in the system. Thus, the OS kernel wouldn't have to change ownership back and forth. For non-kernel OS modules that run in user mode, they could run directly in the application's task/process; this means that there still wouldn't be any ownership change.
Correct. The memory in use would be jointly owned by the kernel and the applications permitted to run in that space. This allocation only changes when you close an application or give a new application rights to use that space.
No, Hans_ is confusing the issue.
If you allow a system structure to be shared between apps, then it only takes one app to to accidentally or maliciously destroy that data structure to take that component down and the system is now broken...
But shared memory areas are not a problem, and most OSs do allow a shared memory space, but their use is very carefully defined in the OS design, and nothing mission critical can be put there.
With AmigaOS, you have NO way to monitor what is being passed between programs/system functions... this is a design feature of AmigaOS... it is defined by the way AmigaOS works!
As soon as you add memory protection, the programs and functions will be passing rubbish between each other... and there is nothing you can do about that, because of the design of AmigaOS!!!
If you add MP, you will be altering so much of the system that you will end up with a new operating system that is totally incompatible with AmigaOS 3.1... and if you are going to that much trouble, why not just take an existing system that already has the design features you want...
In 1998 Apple had the same problem, MacOS was not able to have MP or multitasking by design... 6 years of attempting to retrofit them was fruitless... so they brought in a new system and then put an Emulator to run the old Apps... that would have been the only solution for AmigaOS too.
-
Paolone, I don't know if I was meant to be one of the targets of your 'all talk no action' rant or not
Nope. And I've found your proposals for a PM scheme very interesting: at least you're proposing a solution, instead of simply whining 'cos there's no PM in AROS. My rants were almost directed to the initial post.
Thanks for your vision on VmwAROS, I'm really happy it has been well received by the community. But VmwAROS is in first place an example of my "take action" motto: I felt AROS needed a modern, complete distribution since AROS-MAX was dead, and I simply MADE it.
I'm neither a C developer nor a IT graduate, I'm simply what someone would call "power user". VmwAROS demonstrates that everyone of us, instead of simply whining or struggling in a stupid poor men's war, has the strenght to actually do something. It requires just a lot of will and some spare time.
-
Hans_ wrote:
@SamuraiCrow
I think the answer to the issues that are often associated with memory protection are generally better addressed by using "safe" programming languages like Oberon and some versions of Basic that either don't directly support pointers in general, or have tight limitations on the use of arrays and other data structures to monitor them for overruns and NULL pointer values internally.
The problem with that is that it requires all software to play by the rules. Malicious code does not. Memory protection helps prevent malicious code from doing harm by restricting access to resources.
Hans
Memory protection doesn't stop malicious code either. In a black-box operating system malicious code can overwrite anything that is in the same memory page as it is. In a white-box operating system it can corrupt any system that the code has access to.
The problem with memory protection is that it is limited to memory granularity of the page size of the MMU which is usually 4k. If the assign command is only a couple hundred bytes long and you add memory protection to it it becomes 4k for the code, 4k for the stack, and 4k for any global data. That's 12k for the smallest program to run. That eliminates all of the streamlined efficiency of the AmigaOS when it needs more memory than it allocates just to run a small command.
-
SamuraiCrow wrote:
Hans_ wrote:
@SamuraiCrow
I think the answer to the issues that are often associated with memory protection are generally better addressed by using "safe" programming languages like Oberon and some versions of Basic that either don't directly support pointers in general, or have tight limitations on the use of arrays and other data structures to monitor them for overruns and NULL pointer values internally.
The problem with that is that it requires all software to play by the rules. Malicious code does not. Memory protection helps prevent malicious code from doing harm by restricting access to resources.
Hans
Memory protection doesn't stop malicious code either. In a black-box operating system malicious code can overwrite anything that is in the same memory page as it is. In a white-box operating system it can corrupt any system that the code has access to.
It's no panacea, but it does restrict the amount of damage that can be done. Private memory belonging to multiple processes shouldn't be on the same page. Using more memory than necessary for tiny programs is simply a price that you have to pay.
Let me put it this way, while programming languages/compilers could theoretically generate safe code, there's nothing to stop someone from deliberately generating nasty code. Being able to compromise the whole system with a few trivial instructions is not acceptable. Sure, you may say, just don't run code from people you don't trust. Ok, how do you know who to trust? Protecting the system is part of a modern OS' job.
Hans
-
@bloodline
Whilst thinking about how we would need to change the core of AmigaOS to allow for MP, I thought of a talking point we can use to (hopefully) progress our discussion into something more productive. Am I right in thinking that your main argument behind 'no MP' with the OS3.1 API is because you can't think of a way of adding resource tracking without breaking compatibility?
If I am correct, then lets look at how API calls with MP and non-MP operating systems compare. Lets take the allocation of memory API call for example. In AmigaOS we have AllocMem:
http://utilitybase.com/ref/?keyword=AllocMem&funcgroup=AmigaOS&action=Search
"memoryBlock = AllocMem(byteSize, attributes)"
Now let's look at malloc, the equivalent memory allocation function in FreeBSD, an OS with MP:
http://www.freebsd.org/cgi/man.cgi?query=malloc&sektion=9
"malloc(unsigned long size, struct malloc_type *type, int flags);"
Okay, so let's compare these API calls. First we can see they both have attributes that allow you to set the size of memory space required. They also both have flags/attributes you can set to determine the behaviour of that memory space. With AllocMem (AmigaOS), we have the following flags:
MEMF_CHIP (request ChipMem), MEMF_FAST (request FastMem), MEMF_LOCAL (memory survives CPU reset), MEMF_24BITDMA (used for setting memory for Zorro devices), MEMF_KICK (allows memory to be accessed by Exec early in boot process), MEMF_CLEAR (sets memory to all 'zeros'), MEMF_REVERSE (allocate from top of memory pool), MEMF_NO_EXPUNGE (prevents memory from being expunged on failed memory allocation)
Now let's look at the flags we can set using malloc (FreeBSD):
M_ZERO (sets memory to all 'zeros'), M_NOWAIT (returns value if memory cannot be set immediately), M_WAITOK (indicates okay to wait for memory to be set), M_USE_RESERVE (indicates memory can be set in memory reserved for OS use)
Take a good look at AllocMem and malloc. If I didn't tell you that one function was used on a non-MP OS and the other was on a MP OS, would you have been able to tell just by looking at the API call structure? No.
Similarly, the API call itself doesn't care if it's running on a MP OS or non-MP OS. There is no 'anti-MP' flag for AllocMem. Resource tracking is a separate issue from API calls, it is what the kernel handles.
Thoughts?
-
HenryCase wrote:
@bloodline
Whilst thinking about how we would need to change the core of AmigaOS to allow for MP, I thought of a talking point we can use to (hopefully) progress our discussion into something more productive. Am I right in thinking that your main argument behind 'no MP' with the OS3.1 API is because you can't think of a way of adding resource tracking without breaking compatibility?
No, not at all. AROS in fact implements resource tracking.
If I am correct, then lets look at how API calls with MP and non-MP operating systems compare. Lets take the allocation of memory API call for example. In AmigaOS we have AllocMem:
http://utilitybase.com/ref/?keyword=AllocMem&funcgroup=AmigaOS&action=Search
"memoryBlock = AllocMem(byteSize, attributes)"
Now let's look at malloc, the equivalent memory allocation function in FreeBSD, an OS with MP:
http://www.freebsd.org/cgi/man.cgi?query=malloc&sektion=9
"malloc(unsigned long size, struct malloc_type *type, int flags);"
Okay, so let's compare these API calls. First we can see they both have attributes that allow you to set the size of memory space required. They also both have flags/attributes you can set to determine the behaviour of that memory space. With AllocMem (AmigaOS), we have the following flags:
MEMF_CHIP (request ChipMem), MEMF_FAST (request FastMem), MEMF_LOCAL (memory survives CPU reset), MEMF_24BITDMA (used for setting memory for Zorro devices), MEMF_KICK (allows memory to be accessed by Exec early in boot process), MEMF_CLEAR (sets memory to all 'zeros'), MEMF_REVERSE (allocate from top of memory pool), MEMF_NO_EXPUNGE (prevents memory from being expunged on failed memory allocation)
Now let's look at the flags we can set using malloc (FreeBSD):
M_ZERO (sets memory to all 'zeros'), M_NOWAIT (returns value if memory cannot be set immediately), M_WAITOK (indicates okay to wait for memory to be set), M_USE_RESERVE (indicates memory can be set in memory reserved for OS use)
Take a good look at AllocMem and malloc. If I didn't tell you that one function was used on a non-MP OS and the other was on a MP OS, would you have been able to tell just by looking at the API call structure? No.
Similarly, the API call itself doesn't care if it's running on a MP OS or non-MP OS. There is no 'anti-MP' flag for AllocMem. Resource tracking is a separate issue from API calls, it is what the kernel handles.
Thoughts?
Ok, I can see where you are missing the problem I think.
The problem is not that you can't add MP to AmigaOS. The problem is that when you add it, the system doesn't work like AmigaOS any more.
In an MP OS, you pass data between programs and system functions. This is how they communicate. This is clearly defined behavior, as the OS and the programs "know" that they all live in separate address spaces... which are to all intents and purposes separate computers.
In AmigaOS, programs pass pointers, NOT DATA, to each other... these pointers reference memory anywhere in the flat address space. As soon as each task gets it's own address space, the pointers which are passed between programs are meaningless!!! Thus all amiga software stops working! There is no way to fix this since AmigaOS only has one message passing system and the OS puts no controls on what is passed.
-
As an example,
Two task in AmigaOS... A and B
Task A will allocate some memory and fill it with some Data, and then send a pointer to Task B. Task B receives that pointer and uses it to access the Data put there by Task A. This is a very quick process and requires only the memory allocated by Task A.
In an MP OS there are several strategies used to pass data, I will only present a simple one here.
Task A will allocate some memory and fill it with Data. It then must actually send that Data to task B, this means physically copying the Data from Task A's address space to Task B's Address space byte by byte...
there are actually other strategies that one might employ (since this is rather slow) but it shows clearly the problem which we have to overcome!
Task B must have it's own copy of the same Data in it's own address space in order to work on it.
Amiga OS can happily use the slower more complex strategies for passing data, which are MP compatible... but these methods were never required and thus not used either by the OS or any applications... so we are stuck with a non MP compatible OS... since there is no way to know what the OS and task are passing to each other!
-
In your AllocMem() Vs malloc() example...
The difference is this;
AllocMem() returns an address that points to the same bit of memory to any program that were to receive that pointer.
malloc() returns an address that is meaningful only to the task that called the function... any other task that receives that pointer has just received a meaningless pointer which points to nowhere that it can address!
-
Hi guys,
sorry I didn`t read the whole thread...but here some notes from me. I suppose MP in this context means Multiprocessing, right?
It`s true that most comercial and free operating systems out there use a separate address space for each process. This is a design decision** and not mandatory nor necessary for a MP capable OS. It`s possible to have MP working on a single address space and to use the MMU to extend this space beyond the physical memory by using the page swaping techique and to handle memory protection. Shared resources must still be protected through mutexes or semaphores from concurrent CPU accesses. In a single (flat) address space OS (SASOS) message passing is done by passing pointers to the data in memory, just like AmigaOS does, there is no need to move or copy data around. Note that even in non-SASOS data is generally not copied around, it`s mapped with the help of the MMU, so both processes may see the same data in the same place or in case a process already allocated that virtual memory region for other stuff, in different memory places.
** in the early days of computing when there was 16 bit address spaces only, address translations didn`t make much sense because one could not get over the 16 bit address space with it. Earlier MMUs just swapped memory regions with the bank switching method, thus having many code blocks using the same adresses but only one at a time. In the 32 bit era this technique became obsolete and segmenting and paging took over. However, using for each process different address spaces mapped to the same region simplified program loading as programs could be linked statically and did not need to be relocated in memory. In the 64 bit era there is realy no need to have many address spaces any more. Newer research OSes (e.g. singularity from Microsoft) are in fact SASOS.
Best regards,
Leo24
-
Leo24 wrote:
Hi guys,
sorry I didn`t read the whole thread...but here some notes from me. I suppose MP in this context means Multiprocessing, right?
No, we are talking about Memory Protection.
Lets not even go near Multiprocessing yet as that is a whole other issue, which AmigaOS isn't well suited to...
It`s true that most comercial and free operating systems out there use a separate address space for each process. This is a design decision** and not mandatory nor necessary for a MP capable OS. It`s possible to have MP working on a single address space and to use the MMU to extend this space beyond the physical memory by using the page swaping techique and to handle memory protection. Shared resources must still be protected through mutexes or semaphores from concurrent CPU accesses. In a single (flat) address space OS (SASOS) message passing is done by passing pointers to the data in memory, just like AmigaOS does, there is no need to move or copy data around. Note that even in non-SASOS data is generally not copied around, it`s mapped with the help of the MMU, so both processes may see the same data in the same place or in case a process already allocated that virtual memory region for other stuff, in different memory places.
AmigaOS tasks often rely on a Forbid() halting all multitasking and that a high priority task WILL block a lower priority one... on a Multiprocessor system this condition may not always be true.
** in the early days of computing when there was 16 bit address spaces only, address translations didn`t make much sense because one could not get over the 16 bit address space with it. Earlier MMUs just swapped memory regions with the bank switching method, thus having many code blocks using the same adresses but only one at a time. In the 32 bit era this technique became obsolete and segmenting and paging took over. However, using for each process different address spaces mapped to the same region simplified program loading as programs could be linked statically and did not need to be relocated in memory. In the 64 bit era there is realy no need to have many address spaces any more. Newer research OSes (e.g. singularity from Microsoft) are in fact SASOS.
Best regards,
Leo24
-
@HenryCase
an you talk me through the way OpenWindow() is used in OS 3.1 so that I can look at creating this separation based on how it actually works and not my basic interpretation of Amiga API guides from the Internet.
Uhm... You can not learn it from API guides. Essentially because many actions are not done through API. For example windows can have UserPort. This is allocated by operating system or by application and sometimes there is no UserPort at all. Sometimes application creates its own UserPorts and pokes this into struct Window and Intuition starts using it when you call ModifyIDCMP(). To make things even more complex it is perfectly legal that application A opens a window, application B supplies UserPort to that window and application C manages this window (maybe drawing something). And then there is Intuition task called input.device which manages all windows. You can not assume anything about ownership of that window and its associated structures.
Similarly you can not know who is really going to use memory allocated by AllocMem() or malloc() (in the end malloc() calls AllocMem()). On Amiga you can share all memory allocated by AllocMem()/malloc(), you can even share your code and structures allocated from the stack. There is no absolutely privacy on Amiga. Everything you do is exposed to other applications and both applications and kernel exploit this.
Especially Kickstart 2.0 added many aspects to API which make adding MP even more difficult than it was for Kickstart 1.3.
I should also remind you that OpenWindow() is only one of those few hundred functions which should change to introduce memory protection. Even message passing system in Amiga is flawed in this sense.
To put it short Amiga API is completely broken and can not get fixed. If you want MP, that is :)
-
@bloodline
No, not at all. AROS in fact implements resource tracking.
Does it also work? :-)
-
itix wrote:
@bloodline
No, not at all. AROS in fact implements resource tracking.
Does it also work? :-)
Not really... :-D
-
@bloodline
I'm glad we're getting somewhere now, you were right I was unaware of the pointer usage in AmigaOS.
bloodline wrote:
As an example,
Two task in AmigaOS... A and B
Task A will allocate some memory and fill it with some Data, and then send a pointer to Task B. Task B receives that pointer and uses it to access the Data put there by Task A. This is a very quick process and requires only the memory allocated by Task A.
Then what we would need for MP would be a way of sticking the OS in-between all communications between tasks by manipulating the pointer data. First we need a way of flagging up when a pointer has been set by a program. I've used this guide to give me an overview of how pointers work:
http://www.codeproject.com/KB/cpp/pointers.aspx
So in C you set the pointer variable with a command like this:
int* pNumberOne; (for a pointer of integer type)
Now going to compare this with Pascal. Using this guide...
http://www.learn-programming.za.net/programming_pascal_learn13.html
So if you wanted to declare the same pointer in Pascal you would use these commands:
var
pNumberOne: ^integer;
My question is this. When it comes to compiling the C and Pascal programs will they have the same machine code value for the pointer we declared?
-
@HenryCase
You can't in any sensible way intercept the pointer usage, so the question is irrelevant. You can't "flag" or any other way know when the app is using pointers and/or if it's using pointers correctly or not.
You can't have MP with AmigaOS, get over it.
The best limited MP hack for AmigaOS is MMULib (http://aminet.net/package/util/libs/MMULib) by Thomas Richter. It has a MuLink tool that can write protect certain segments of the executables. Obviously the program in question must be compiled to take advantage of this (basically only code and read only data can be protected etc). This is far from full MP however.
-
this is really easy, focus on the important x86 version of aros, after all it was there it all started. the rest of the aros ports is of no important at the moment. finish x86 aros with everything that is necessary, like an amiga emulator integrated for classic games and software and firefox web browser...
-
Piru wrote:
@HenryCase
You can't in any sensible way intercept the pointer usage, so the question is irrelevant. You can't "flag" or any other way know when the app is using pointers and/or if it's using pointers correctly or not.
No sensible way intercept pointer usage? This is linked to the 'can it be done elegantly' question I mentioned previously, whereas this discussion is still at the 'can it be done' stage. Even if the MP solution we find is impractical, but works, then at least we'll know what our real options are.
So I'll ask again: is it possible to write a program that can scan the binary of another application and know when a pointer has been set?
-
HenryCase wrote:
@bloodline
I'm glad we're getting somewhere now, you were right I was unaware of the pointer usage in AmigaOS.
bloodline wrote:
As an example,
Two task in AmigaOS... A and B
Task A will allocate some memory and fill it with some Data, and then send a pointer to Task B. Task B receives that pointer and uses it to access the Data put there by Task A. This is a very quick process and requires only the memory allocated by Task A.
Then what we would need for MP would be a way of sticking the OS in-between all communications between tasks by manipulating the pointer data. First we need a way of flagging up when a pointer has been set by a program.
There is no way to know what the programs are passing to each other. If Task A sends a number to Task B, the OS has no way of knowing how Task B will use the number! That number could be a pointer, or it could be some data... there is no difference technically, to the OS and CPU it is just a number.
AmigaOS simply does not have any mechanism, and worse, never has had any provision for identifying what is being passed between programs.
I've used this guide to give me an overview of how pointers work:
http://www.codeproject.com/KB/cpp/pointers.aspx
So in C you set the pointer variable with a command like this:
int* pNumberOne; (for a pointer of integer type)
Now going to compare this with Pascal. Using this guide...
http://www.learn-programming.za.net/programming_pascal_learn13.html
So if you wanted to declare the same pointer in Pascal you would use these commands:
var
pNumberOne: ^integer;
My question is this. When it comes to compiling the C and Pascal programs will they have the same machine code value for the pointer we declared?
As I keep saying in every post, you are thinking high level... the Computer does not work at this high level... there is no difference at a machine level between a number and a pointer!
-
@HenryCase
So I'll ask again: is it possible to write a program that can scan the binary of another application and know when a pointer has been set?
And I answer again: No.
-
HenryCase wrote:
Piru wrote:
@HenryCase
You can't in any sensible way intercept the pointer usage, so the question is irrelevant. You can't "flag" or any other way know when the app is using pointers and/or if it's using pointers correctly or not.
No sensible way intercept pointer usage?
No way, period. You could make assumptions but at best you would only be right, less than 30% of the time.
This is linked to the 'can it be done elegantly' question I mentioned previously, whereas this discussion is still at the 'can it be done' stage. Even if the MP solution we find is impractical, but works, then at least we'll know what our real options are.
It can't be done... No system could be made that would be in anyway deterministic.
So I'll ask again: is it possible to write a program that can scan the binary of another application and know when a pointer has been set?
No. There is nothing special about a pointer... it looks exactly the same as any other kind of number.
-
@bloodline and @Piru
Thank you for your posts, I will now reply to them in one go.
bloodline wrote:
There is no way to know what the programs are passing to each other.
One thing at a time. First we need to know where the pointers are, then I'll say about my ideas on how we can use this information.
Piru wrote:
And I answer again: No.
Please explain the reasoning behind your answer, a simple 'No' doesn't help me understand why you don't think there is a way to see pointers in binary code.
Whilst I wait for your explanation, I would like to ask you another question. How does a CPU know that a program is asking to set up a pointer?
bloodline wrote:
No way, period. You could make assumptions but at best you would only be right, less than 30% of the time.
ASM is closely linked to the structure of machine code, correct? Also, ASM strives for efficiency. I would argue that there are only so many ways to set a pointer in ASM efficiently, and therefore only so many binary structures we would need to scan for to get a higher accuracy than 30% (where you got that figure from I don't know). I found a couple of ways of setting pointers in x86 ASM (I'm assuming it's x86 ASM anyway) from this page (Google cache version):
cs.colgate.edu/faculty/nevison/cs201web/lectureNotes/arraysandpointers.ppt
========================
clear2:
add $t0, $a0, $zero # p = addr of array[0]
forp: sw $zero, 0($t0) # memory[p] = 0
addi $t0, $t0, 4 # p = p + 4
sll $t1, $a1, 2 # $t1 = 4 * size
add $t1, $t1, $a0 # $t1 = addr array[size]
slt $t2, $t0, $t1 # p < &array[size] ?
bne $t2, $zero, forp # if so, continue
endfor:
=========================
Pointer version assembly code
clear2:
add $t0, $a0, $zero # p = addr of array[0]
sll $t1, $a1, 2 # $t1 = 4 * size
add $t1, $t1, $a0 # $t1 = addr array[size]
forp: sw $zero, 0($t0) # memory[p] = 0
addi $t0, $t0, 4 # p = p + 4
slt $t2, $t0, $t1 # p < &array[size] ?
bne $t2, $zero, forp # if so, continue
endfor:
========================
Why is it impossible to scan for the binary versions of the ASM code above?
-
HenryCase wrote:
@bloodline and @Piru
Thank you for your posts, I will now reply to them in one go.
bloodline wrote:
There is no way to know what the programs are passing to each other.
One thing at a time. First we need to know where the pointers are, then I'll say about my ideas on how we can use this information.
You can't say that!
What if the pointer never actually exists anywhere and is calculated by Task A and then passed to Task B?
Piru wrote:
And I answer again: No.
Please explain the reasoning behind your answer, a simple 'No' doesn't help me understand why you don't think there is a way to see pointers in binary code.
I'll answer, again, since I've repeatedly said it before :-)
There is nothing special about a pointer... it is just a number, neither the CPU or the OS has any idea what that number is for.
The CPU just pushes numbers around... buy a good book on 68K ASM and learn the CPU inside out... it shouldn't take more than a few weeks to figure it out... you can't hope to understand this problem until you know a CPU.
Whilst I wait for your explanation, I would like to ask you another question. How does a CPU know that a program is asking to set up a pointer?
There is no "setting up a pointer", it's just a number to the CPU... nothing special.
bloodline wrote:
No way, period. You could make assumptions but at best you would only be right, less than 30% of the time.
ASM is closely linked to the structure of machine code, correct?
ASM is the human readable form of machine code, but often has some high level features to aid coding.
Also, ASM strives for efficiency.
Not really... My ASM code was far from efficient ;-)
I would argue that there are only so many ways to set a pointer in ASM efficiently, and therefore only so many binary structures we would need to scan for to get a higher accuracy than 30% (where you got that figure from I don't know).
My 30% figure was if you tried to assume what the tasks were sending to each other.
It really doesn't matter what a task does in it's own address space. The problems occur when other tasks attempt to access structures which are not in their address space.
Please get over this obsession with pointers... they are not going to solve any problems.
The problems are with the design of AmigaOS... not the CPU or the programming languages used!!
I found a couple of ways of setting pointers in x86 ASM (I'm assuming it's x86 ASM anyway) from this page (Google cache version):
cs.colgate.edu/faculty/nevison/cs201web/lectureNotes/arraysandpointers.ppt
========================
clear2:
add $t0, $a0, $zero # p = addr of array[0]
forp: sw $zero, 0($t0) # memory[p] = 0
addi $t0, $t0, 4 # p = p + 4
sll $t1, $a1, 2 # $t1 = 4 * size
add $t1, $t1, $a0 # $t1 = addr array[size]
slt $t2, $t0, $t1 # p < &array[size] ?
bne $t2, $zero, forp # if so, continue
endfor:
=========================
Pointer version assembly code
clear2:
add $t0, $a0, $zero # p = addr of array[0]
sll $t1, $a1, 2 # $t1 = 4 * size
add $t1, $t1, $a0 # $t1 = addr array[size]
forp: sw $zero, 0($t0) # memory[p] = 0
addi $t0, $t0, 4 # p = p + 4
slt $t2, $t0, $t1 # p < &array[size] ?
bne $t2, $zero, forp # if so, continue
endfor:
========================
Why is it impossible to scan for the binary versions of the ASM code above?
This looks like PPC asm... which I really don't like and was never meant for humans to read... but you miss the point, the CPU doesn't see the high level part of the code. it just adds... subtracts... shifts... ands... ors... nots... numbers! It really doesn't have any idea what the programmer is trying to do, it can't know, and nothing outside of the CPU can know either!
What you are presenting here is totally irrelevant to the problem, which is actually very simple. AmigaOS doesn't know what programs are saying to each other or how programs interact with each other.... AmigaOS allows the programmer to make the programs work in anyway he chooses, that is a design feature of AmigaOS to save memory and be quick on a 7Mhz CPU... Remove that feature and nothing written for AmigaOS works anymore.
Once you add MP, you have to impose restrictions on the programmer and provide him with clearly defined interfaces and methods for achiving a task.
-Edit- I want to add that the pointer problem is deeper than just sharing Data... Code is also shared on AmigaOS, the entire library system is built around the idea than one executable and jump into and run code in another executable... something totally alien to a Memory protected OS...
-
@HenryCase
To scan for those binary equivalents you'd have to know the difference between code and data. If you write a patcher to go through and change all of the binary instances of memory access outside the main chunk of code, you'd probably end up inadvertently changing some constant data to something different. In short you'd have to go into every program by hand and write patch code for every instance of pointer access.
Why not just get the source codes of everything you'd ever want to run on AROS and recompile it for MacOSX? You'd save yourself a lot of headaches and the rest of us a lot of hassle. :-x
-
I could go for a port of PrBOOM and zDOOM (if not GzDOOM) while we are waiting on updates. ;)
-
SamuraiCrow wrote:
@HenryCase
To scan for those binary equivalents you'd have to know the difference between code and data. If you write a patcher to go through and change all of the binary instances of memory access outside the main chunk of code, you'd probably end up inadvertently changing some constant data to something different.
I would actually be surprised if you even could identify the memory access, you have no way to know what and where that pointer is going... thus why a JIT can only translate code blocks up to the jumps.
In short you'd have to go into every program by hand and write patch code for every instance of pointer access.
And even if you could do that, it doesn't solve the problem... AmigaOS programs are designed at a fundamental level to interact intimately, you would have to rewrite the program completely.
Why not just get the source codes of everything you'd ever want to run on AROS and recompile it for MacOSX? You'd save yourself a lot of headaches and the rest of us a lot of hassle. :-x
I agree... but you'd have a hell of a time porting a program from AmigaOS to MacOSX... I would rather just rewrite the program.
-
XDelusion wrote:
I could go for a port of PrBOOM and zDOOM (if not GzDOOM) while we are waiting on updates. ;)
AROS has both Doom and Quake ports, that work really well... :-)
-
bloodline wrote:
AROS has both Doom and Quake ports, that work really well... :-)
And it would be really fine if someone would look at their sources, and add AHI audio support.
-
@HenryCase
The OS4 development team have looked at this, and decided that the only way to make it happen would be to break compatibility. Piru and Bloodline say so too. Trying to somehow discover who owns what memory, and who's still using it would be way to costly computationally, even if you could do it. The correct way forward is to create stricter rules about sharing memory, and that can't be done without breaking compatibility.
Whether it's AOS4.x, MOS, or AROS, the only way to get full memory protection would be to modify the API in a manner that breaks compatibility. Old apps would be run in a sand-boxed environment.
I'd be happy if they did break compatibility and provide a sand-box for old apps. While they're doing that, they might as well break other stuff to allow SMP, and to lock off user programs from accessing OS structures.
Hans
-
paolone wrote:
1. just whining and claiming their opinion would be the best way to improve AROS
2. contribute with code to reach the common goal stated above.
3. take the existing code and improve upon it the way they like, just to become the new ruler of the world.
The direction is already here for the people that want to see it. The others may disagree, but please don't claim AROS is pointless or directionless anymore, as it is completely untrue.
How about adding:
4. If you can't program, donate to an AROS bounty, even if $1.
-
"I'd be happy if they did break compatibility and provide a sand-box for old apps. While they're doing that, they might as well break other stuff to allow SMP, and to lock off user programs from accessing OS structures. "
I agree with you. Of course this brings the conversaton back to the forking argument. The point of not breaking compatibility is so old software could continue to run. Right now, we have to rely on AInc to run most old software because they own the Kickstarts. My primary interst in AROS is that I keep hoping that it will eventually get reported to the 68k, and we could finally get rid of AInc.
I guess the way I see it is that for any old software, I want to be able to run it on my MiniMig, one of it's successors, and maybe UAE. For new software, it doesn't matter if we break compatibility with the old API.
To answer the original questions:
1. Can it be done?
Yes.
2. Can it be done elegantly?
Sure, if Aros was ported to 68k so that we don't have to go to AInc to get kickstarts, and UAE were integrated so that it worked out of the box using that port it be plenty elegant.
3. Should it be done?
That is the question. I would say it would make sense, but that the 68k port is far more important than it is given credit for.
-
@HenryCase
I want to apologize to had pbrought the old discussion out, if only i knew was going to do such a mess, but maybe is better this way, so the ideas were put under verification.
Actually from my point of view i might be simply appy if the modifications made by Michael were ported to the base x86 build; i know through the developer mailing list that staf is rewriting the ABI therefore, considered that the programs made so far will require a rewrite anyway, better to include even those modifications to the tree.
@FairlaneFastback
Is my intention to contribute to some of the bounties, in particular would like to see a bounty to bring some basic office application on AROS as siag office or similar and guess who else might be interestedin such a bounty....
Saimon69
-
@all
Thank you for your posts.
@bloodline and @SamuraiCrow
I would like to pick up on one point which has been stated by you both in the past, which is that there isn't a way to distinguish between instructions and data. In my admittedly limited understanding of the subject that seems like complete tosh, so forgive me whilst I persist with my questioning to discover who is mistaken.
According to Wikipedia, the following four steps are used in classical CPU architecture: fetch, decode, execute, writeback. The second step 'decode' is of interest to us here. Again, according to Wikipedia:
"In the decode step, the instruction is broken up into parts that have significance to other portions of the CPU. The way in which the numerical instruction value is interpreted is defined by the CPU's instruction set architecture(ISA). Often, one group of numbers in the instruction, called the opcode, indicates which operation to perform. The remaining parts of the number usually provide information required for that instruction, such as operands for an addition operation."
So for even the most basic program to function the CPU must know the difference between the opcode part of the instruction and the operand part of the instruction. Can we agree on that at least?
saimon69 wrote:
@HenryCase
I want to apologize to had pbrought the old discussion out, if only i knew was going to do such a mess, but maybe is better this way, so the ideas were put under verification.
No worries mate, I'm glad you did bring back this old discussion as it has been helpful for me.
saimon69 wrote:
i know through the developer mailing list that staf is rewriting the ABI
I took myself off the AROS mailing list, but I am curious to know how Staf is getting on with writing/rewriting the ABI. Any news you know of on that front?
-
HenryCase wrote:
I would like to pick up on one point which has been stated by you both in the past, which is that there isn't a way to distinguish between instructions and data.
They are right, there isnt.
In my admittedly limited understanding of the subject that seems like complete tosh, so forgive me whilst I persist with my questioning to discover who is mistaken.
You lack a basic fundamental understanding of how a cpu operates, so this thread goes around in circles when people say no, you say yes.
According to Wikipedia, the following four steps are used in classical CPU architecture: fetch, decode, execute, writeback. The second step 'decode' is of interest to us here. Again, according to Wikipedia:
.....
So for even the most basic program to function the CPU must know the difference between the opcode part of the instruction and the operand part of the instruction. Can we agree on that at least?
The cpu does know the difference between the opcode bits and other bits of the bytes it loads. The problem is that the cpu just loads a stream of bytes into its decoder. it looks at the bits and says mmm these 5 bits mean xor... I'll go run the XOR opcode. it doesnt know what its decoded is a string of text. It loads whatever the instruction pointer is pointing at. If things get really bad, it throws an exception.
-
yakumo9275 wrote:
You lack a basic fundamental understanding of how a cpu operates, so this thread goes around in circles when people say no, you say yes.
The idiot (like me) knows no limits, which is a strength as well as a weakness.
The cpu does know the difference between the opcode bits and other bits of the bytes it loads. The problem is that the cpu just loads a stream of bytes into its decoder. it looks at the bits and says mmm these 5 bits mean xor... I'll go run the XOR opcode. it doesnt know what its decoded is a string of text. It loads whatever the instruction pointer is pointing at. If things get really bad, it throws an exception.
"I'll go run the XOR opcode", so in other words I'm right.
The CPU has the ability to recognise an XOR opcode, and every other opcode it uses, and if part of an instruction isn't an opcode it is data. Therefore a CPU knows, after the 'decode' process, what is an opcode and what is an operand, correct? This requires just a yes or no answer.
-
When the CPU reads a memory location or a block of memory, it fetches something like this:-
1011001 11001100 11101010 00110110 01101111 00001110
the only way to interpret this "data" is to know what the CPU is expecting.
HenryCase wrote:
I would like to pick up on one point which has been stated by you both in the past, which is that there isn't a way to distinguish between instructions and data.
Every program has a start point and the CPU knows this start address WILL be an instruction (opcode) but the second byte maybe another opcode or it could be data.
According to Wikipedia, the following four steps are used in classical CPU architecture: fetch, decode, execute, writeback. The second step 'decode' is of interest to us here. Again, according to Wikipedia:
.....
So for even the most basic program to function the CPU must know the difference between the opcode part of the instruction and the operand part of the instruction. Can we agree on that at least?
So the only way for the operating system to know if a byte contains an opcode or data is to emulate the cpu in software and follow the execution of the program from the start, which will be very slow, it cannot simply read a byte and know whether it has fetched an opcode or data.
-
HenryCase wrote:
"I'll go run the XOR opcode", so in other words I'm right.
The CPU has the ability to recognise an XOR opcode, and every other opcode it uses, and if part of an instruction isn't an opcode it is data. Therefore a CPU knows, after the 'decode' process, what is an opcode and what is an operand, correct? This requires just a yes or no answer.
Sure it knows, this is how emulators like EUAE run.
x86 for example, uses all 256 'bytes' as master opcodes. That means 'A' is an opcode and 'B' and 'C' and 'a' and 'q' etc. EG: "Edgar Allan Poe" is it text or is it data? Could be valid as both, or one or the other. You cant arbitrarily lable it as either.
Back to your example, say on powerpc, XOR is encoded with 5 bits, so after the 'decode' process, the 5bit XOR opcode is out, the processor is left with 26bits.. only they are not data, its still the same opcode. (all ppc instructions are 32bits).
The problem is, code and data are ambiguous, they are the same thing. There is no rule that says code can not be data or data can not be code in a binary image. The processor _assumes_ its code.
I can see this is fruitless so I'm going to stop replying. I hereby agree with all you say, you are correct, we are all ignorant. Yes memory protection is so easy to add to amiga os, we can trap pointers, detect valid opcodes in a sea of data, magically understand self modifying code and can beat gary kasparov at mahjong with our eyes closed!
I'm done, stick a fork in me.
-
HenryCase wrote:
"I'll go run the XOR opcode", so in other words I'm right.
The CPU has the ability to recognise an XOR opcode, and every other opcode it uses, and if part of an instruction isn't an opcode it is data. Therefore a CPU knows, after the 'decode' process, what is an opcode and what is an operand, correct? This requires just a yes or no answer.
The problem is that your data may just happen to match an op-code. Op-codes are restricted in number, but data can take any value. Thus, if you recognize an XOR op-code in memory, it could be an op-code, but it might not.
You could also check whether the memory is marked as executable, in which case you could probably confirm that something is indeed an op-code. Now what? That doesn't actually tell you what it's doing with data, who the data belongs to, etc.
Hans
-
yakumo9275 wrote:
Yes memory protection is so easy to add to amiga os
I never said it was easy, or even desirable, but I do think it can be done. What's wrong with wanting to discuss that?
Anyway the rest of your post before you went doolally was useful, so thank you.
-
@Hans
I'll reply to you soon, just off for some food.
@A6000
A6000 wrote:
When the CPU reads a memory location or a block of memory, it fetches something like this:-
1011001 11001100 11101010 00110110 01101111 00001110
the only way to interpret this "data" is to know what the CPU is expecting.
I think I have a way around this...
A6000 wrote:
So the only way for the operating system to know if a byte contains an opcode or data is to emulate the cpu in software and follow the execution of the program from the start, which will be very slow, it cannot simply read a byte and know whether it has fetched an opcode or data.
Bingo. CPU emulation is part of my plan. Yes it is too slow to use for running programs in real time, but it can be used to construct a 'map' of the program. This 'map' would only need to be created once, but would then be used by the operating system to help guide the program execution at key points.
-
@Hans
Hans wrote:
The problem is that your data may just happen to match an op-code. Op-codes are restricted in number, but data can take any value. Thus, if you recognize an XOR op-code in memory, it could be an op-code, but it might not.
Going back to something yakumo9275 mentioned in his last post:
"XOR is encoded with 5 bits, so after the 'decode' process, the 5bit XOR opcode is out, the processor is left with 26bits.. only they are not data, its still the same opcode. (all ppc instructions are 32bits)."
Do 68k processors have the same 'fixed length op-code' trait? If so, couldn't you read the first how ever many bits (16? 32?) after the program counter in the CPU incremented as the opcode?
Hans wrote:
You could also check whether the memory is marked as executable, in which case you could probably confirm that something is indeed an op-code.
That sounds promising.
Hans wrote:
Now what? That doesn't actually tell you what it's doing with data, who the data belongs to, etc.
Don't opcodes always have the same operand requirements ('x' many operand bits for one opcode format, 'y' many operand bits for another format)? If so, couldn't you use this format to determine what the data used by an operand represented in non-binary form?
-
HenryCase wrote:
Hans wrote:
Now what? That doesn't actually tell you what it's doing with data, who the data belongs to, etc.
Don't opcodes always have the same operand requirements ('x' many operand bits for one opcode format, 'y' many operand bits for another format)? If so, couldn't you use this format to determine what the data used by an operand represented in non-binary form?
No. You could perform an XOR on any type of data, from text, through to graphics, and even pointers.
Hans
-
@Hans_
Thanks for the quick reply.
Hans_ wrote:
No. You could perform an XOR on any type of data, from text, through to graphics, and even pointers.
Sorry, I worded my question badly, I meant more along the lines of the length of the data. So for instance certain addition instructions would have two blocks of data (i.e. the two values to be added together). Is there a way of determining the data length in terms of these blocks?
-
HenryCase wrote:
The CPU has the ability to recognise an XOR opcode, and every other opcode it uses, and if part of an instruction isn't an opcode it is data. Therefore a CPU knows, after the 'decode' process, what is an opcode and what is an operand, correct? This requires just a yes or no answer.
No. Here's an example to make this very straightforward. Assume a mythical computer with an instruction set containing ADD, LOAD, and STORE. ADD sums two registers and puts its output in the first register. We assign these instructions numbers, 0 through 3. Our computer has only two registers, which we'll number 0 and 1. Our computer uses 4 bit 'bytes' (to keep the math simple for me) so we have possible values in each 'byte' of 0 through 15, or 0xF. Now, let's say I have 20 'bytes' of memory starting at 0x00. OK? This is actually a fairly reasonable computer to use as an example here. It's missing branching and comparisons, but they will fit within the model, but they're not needed for this example.
Suppose I want to write a program to add three numbers (3, 5, and 2) without carry, it might look something like this, assuming the three numbers are in the first 3 'bytes' of memory already and I want to output to fourth 'byte':
LOAD reg0, 0x0
LOAD reg1, 0x1
ADD
LOAD reg1, 0x2
ADD
STORE reg1, 0x3
now this assembles to:
0x10011101120313
that's our actual binary executable for our mythical computer there.
Now, suppose I write a program (for a version of this computer with more memory and some I/O capability) which stores a user password. This will be done by using two four-bit 'bytes' per character, so we'll use ASCII. Only we're going to use ROT-n to encode the passwords so they can't be as easily read. Arbitrarily, the program has selected -21 for the offset, n, so each letter has 21 subtracted from it's ASCII value before storage. So if the user enters password "#l|l}n~" we get 0x10011101120313 as the encoded password.
Yes, that's the same binary code as the program. So which is which? Which one will the program execute? Both, and with identical results either way.
Real software will unfortunately have runs of bytes where there is a mix of program code and data. So you can't tell what's code, what's data, and what might be a special kind of data called a pointer.
The _only_ way to implement memory protection for a system where programs read and write shared memory is to run each program in a virtual machine where it is the only application running, like Apple did with the Classic environment in OS X, or to write each and every program that will run on the system to inform the operating system of its intention to use shared memory. Amiga software wasn't written that way, so that options doesn't work.
-
@HenryCase
In a 68k instruction set (as well as the x86), there are variable widths for instructions to have. This would make PowerPC much easier since all of its instructions are the same length.
-
@pkillo
Thanks for your examples, I enjoyed following them. In your imaginary computer how does the CPU know when to increase the Program Counter?
@SamuraiCrow
I see. Well a fixed instruction width would have made our lives easier, but seeing as the vast majority of Amiga software is compiled for 68k CPUs let's see what we can do with that.
-
Variable length instructions are no problem, they are fully documented, if the CPU expects the next byte to be an opcode, that opcode will be decoded and reveal if the following byte is also an opcode.
I think 68k instructions range from 1 to 16 bytes including operands where applicable.
-
Being designed for a 16-bit bus, the 68000 has only 16-bit multiples in its instruction set. That makes all of the opcodes either 16,32,48 or 64-bit sizes. No 8-bit opcodes, in fact, word misalignment will cause a lock up on 68000 series processors.
-
HenryCase wrote:
@pkillo
Thanks for your examples, I enjoyed following them. In your imaginary computer how does the CPU know when to increase the Program Counter?
The program counter is automatically incremented after every instruction execution.
One fly in the ointment for your idea of tracing program execution is that data may not be static, it might be input in real time to change program flow, or data might come from another part of the program which only executes if certain conditions are met, dependant on real world input, or data may come from another program entirely.
You may need to log all input data just in case the program you are tracing needs the data that came in during the trace time. it's all getting quite complicated.
You may not know the data found at the memory location is invalid.
-
A6000 wrote:
One fly in the ointment for your idea of tracing program execution is that data may not be static, it might be input in real time to change program flow, or data might come from another part of the program which only executes if certain conditions are met, dependant on real world input, or data may come from another program entirely.
You may need to log all input data just in case the program you are tracing needs the data that came in during the trace time. it's all getting quite complicated.
You may not know the data found at the memory location is invalid.
A6000, this is the one issue I am trying to resolve at the moment. I'd much rather use some sort of feedback loop system that matches input, output and possibly the emulated CPU state, and analyse this information as a whole, rather than keep track of all inputs and work out the progression as a sequence.
I won't ask for 'any ideas' on this matter as I know I'll just get the usual 'it can't be done' stuff, and I'd much rather have a better explanation than that.
-
Just checked, 68k instructions range from 2 to 10 bytes, though I vaguely recall that instructions relating to co-processors such as the FPU may use up to 16 bytes.
-
HenryCase wrote:
I won't ask for 'any ideas' on this matter as I know I'll just get the usual 'it can't be done' stuff, and I'd much rather have a better explanation than that.
I don't know enough about programming to say it is impossible, but I can see it won't be easy.
Sandboxes have been mentioned several times, maybe this would be easier.
And just how important is memory protection anyway? we managed without it for a quarter of a century.
-
The CPU has the ability to recognise an XOR opcode, and every other opcode it uses, and if part of an instruction isn't an opcode it is data. Therefore a CPU knows, after the 'decode' process, what is an opcode and what is an operand, correct? This requires just a yes or no answer.
It is not so simple. You must actually execute code to find out what is actually data and what is an opcode. It is not big problem, UAE does that when translating memory accesses to local memory space.
But this is not going to help you at all. Just like UAE, 68k processor (virtual or real one) does not care or know about operating systems. It is very difficult to find out when program is calling OpenWindow() and it is even more difficult to find out when that particular function returns.
You should try to disassemble some Amiga program and try to figure out where it is calling OpenWindow()... or figure out memory location where it stores struct Window.
-
@A6000 and @SamuraiCrow
Thank you both for the 68k opcode information. My next question is linked to what you both said last. If 68k opcodes are 2-10 bytes in length, and one opcode can be directly followed by another opcode (i.e. no data in between) how does the instruction code tell the CPU to expect an opcode rather than operand data? Is there an opcode 'flag' instruction, or is it purely determined by the current opcode in use (and its operand needs)?
-
itix wrote:
You should try to disassemble some Amiga program and try to figure out where it is calling OpenWindow()... or figure out memory location where it stores struct Window.
Sounds like a good challenge.
-
@HenryCase
Bingo. CPU emulation is part of my plan. Yes it is too slow to use for running programs in real time, but it can be used to construct a 'map' of the program. This 'map' would only need to be created once, but would then be used by the operating system to help guide the program execution at key points.
No, you can't do that. It's not possible to somehow trigger the program to take all possible codepaths. Thus your map will always be incomplete and thus totally useless (and I'm not even going into how you would automagically be able to fix the programs, which you can't either).
As soon as the program takes codepath it didn't while mapping your concept fails.
-
@Piru
As I said before, I am currently looking of ways around this issue. The code map may end up being in a more high level modular structure. Maybe also we keep a record of the code paths we have taken before, and if a new one arises we track its performance, adding to our code map. These ideas are only rough sketches for now, I only realised through this thread that I'd have to deal with pointer issues when retrofitting Amiga operating systems and apps with MP, so forgive me for not going into more detail.
@all
Whilst I carry on sketching out how to track the performance of Amiga programs, I'd like to take the discussion about MP on a simpler route. Seeing as the whole idea of AROS being OS 3.1 compatible is on a source code level (apart from the plans for binary compatibility on the 68k port), let's start discussing how to graft full MP into AROS through program compilers. Would there be any way to make AllocMem (and all other AROS functions) behave as they should without using pointers (maybe using some new 'port' system instead that AROS could flag and control)?
-
HenryCase wrote:
@A6000 and @SamuraiCrow
If 68k opcodes are 2-10 bytes in length, and one opcode can be directly followed by another opcode (i.e. no data in between) how does the instruction code tell the CPU to expect an opcode rather than operand data? Is there an opcode 'flag' instruction, or is it purely determined by the current opcode in use (and its operand needs)?
When the processor completes an instruction it increments the program counter and fetches the next instruction, the word (2 bytes) returned will specify the instruction, if the instruction requires operands the next 2,4 or 8 bytes will tell the CPU where to get it's operands from. these additional bytes will be immediate data, registers or memory locations, source and destination.
-
HenryCase wrote:
@pkillo
Thanks for your examples, I enjoyed following them. In your imaginary computer how does the CPU know when to increase the Program Counter?
Each instruction has the 'side effect' of incrementing the PC by the size of the instruction (in my example 1 byte) and the total size of the operands (0 or 2 bytes in the example).
btw, this program 'tracing' is probably not impossible; it's just that you will generate a geometrically increasing pile of what I'll call garbage data alongside your program's actual data. I call it garbage data because it's not useful for producing the results of the programs run, but only to 'fix' features of the programs being run. what you're seeing as needed for it is the tip of a gigantic iceberg. It would be cheaper in the long run to write all new software or do without memory protection. Much cheaper.
-
@HenryCase
let's start discussing how to graft full MP into AROS through program compilers.
MP is not the task of the compiler.
Would there be any way to make AllocMem (and all other AROS functions) behave as they should without using pointers (maybe using some new 'port' system instead that AROS could flag and control)?
Even if you could, the actual poking can be in the code, not the OS itself. The code needs to be rewritten, compiler can't possibly know how to do that for you.
-
My two cents re: interpretation of binary data: It's possible to interleave opcodes and operands, where one opcode's operand is actually an opcode itself, and a subsequent jump or branch takes advantage of that fact. This was a common dirty hack in MS-DOS software. You can't assume that every byte (or every bit, even) is either an opcode or an operand. It can be both. Or padding. Or a gamma ray induced parity error. Or an abstract picture of someone's grandmother.
Obviously, that complicates any attempt to map an arbitrarily complex sequence of code. If you do figure out a way to do it optimally in polynomial time, kudos. You'll earn a Fields Medal (if you're under 40 years of age), and the Clay Institute will award you 1,000,000 USD for a generalized solution (http://www.claymath.org/millennium/P_vs_NP/). Hell, you'd probably earn a Nobel Prize as well. ;-) For all we know, you could be the one who solves this problem. Good luck! Sincerely.
Trev
-
now this assembles to:
0x10011101120313
that's our actual binary executable for our mythical computer there.
Now, suppose I write a program (for a version of this computer with more memory and some I/O capability) which stores a user password. This will be done by using two four-bit 'bytes' per character, so we'll use ASCII. Only we're going to use ROT-n to encode the passwords so they can't be as easily read. Arbitrarily, the program has selected -21 for the offset, n, so each letter has 21 subtracted from it's ASCII value before storage. So if the user enters password "#l|l}n~" we get 0x10011101120313 as the encoded password.
That would be a cool 'Demo' challenge. Make programs that are actual sentences in ascii, but do when run as code, do something... or nothing, but don't crash.
-
HenryCase wrote:
Bingo. CPU emulation is part of my plan. Yes it is too slow to use for running programs in real time, but it can be used to construct a 'map' of the program. This 'map' would only need to be created once, but would then be used by the operating system to help guide the program execution at key points.
If you could build a map, you wouldn't need Memory Protection. If you can build a map then you already know every possible outcome of running the program. If fact you wouldn't ever need to run the program, instead you would just look at the end of the map and see the outcome of the execution! And when you see it like that you realise that the map would be gigabytes in size for even the simples program.
-
HenryCase wrote:
@Piru
As I said before, I am currently looking of ways around this issue.
First you need to understand the problem... at the moment you are looking at the symptoms of the problem and trying to make the symptoms go away, without actually sorting out the underlying problem.
The code map may end up being in a more high level modular structure.
A map is a map... I don't think you understand the scale of what you are suggesting!
Maybe also we keep a record of the code paths we have taken before, and if a new one arises we track its performance, adding to our code map.
Imagine you are trying to plot the Code path for a game, lets take Quake... you would have to make a code path for every single possible variation in movement made by the player, at every frame of the game... I don't think that you could map that in a thousand years...
How do you deal with prorgams that are entirely intimate... one map for both programs? one map for each program, but which have common parts? How do you deal with the fact that in AmigaOS, everything essentially runs as one big program?
These ideas are only rough sketches for now, I only realised through this thread that I'd have to deal with pointer issues when retrofitting Amiga operating systems and apps with MP, so forgive me for not going into more detail.
A Sketch it may be... but already you should be able to see that it's a flawed idea, and one that actually does nothing to solve our initial problem... which is that the OS has no way to know what a program is up to and hence no way to enforce any restrictions upon those programs.
@all
Whilst I carry on sketching out how to track the performance of Amiga programs, I'd like to take the discussion about MP on a simpler route. Seeing as the whole idea of AROS being OS 3.1 compatible is on a source code level (apart from the plans for binary compatibility on the 68k port), let's start discussing how to graft full MP into AROS through program compilers.
This doesn't make any sense. A compiler cannot add memory protection to a program... Just as a dictionary does not make for good bed time reading.
Would there be any way to make AllocMem (and all other AROS functions) behave as they should without using pointers (maybe using some new 'port' system instead that AROS could flag and control)?
Do you even know what a pointer is? How can AllocMem() work without returing a pointer to the memory you have just allocated? Could you write a story without using verbs?
Have you googled MicroSoft's Singularity yet? I think there is an OS much better suited to your needs than Amiga OS ever will be...
-
@all
I'm at work so I don't have time to reply to all of your points.
@Trev
Trev wrote:
the Clay Institute will award you 1,000,000 USD for a generalized solution (http://www.claymath.org/millennium/P_vs_NP/).
You get 1 million USD for solving that 400 students problem?! I've already started. ;-)
Piru wrote:
The code needs to be rewritten, compiler can't possibly know how to do that for you.
Yes it can. Let's say you rewrite the compiler to add a "-MPAROS" compiler flag. Then whenever you try and compile programs with this flag it follows certain rules to make programs compatible with an MP kernel. The resultant code is different from compiling it normally, but crucially you wouldn't need to rewrite the application code yourself.
-
But henry, you are missing the point. The OS already knows when a message is sent. The OS has quite a bit of information regarding message actually! But the OS does not know what the contents of the message is or how it is to be used... The compiler doesn't know this either.
The problem comes when that message contains references to memory areas not in the context of the receiving task... In AmigaOS this is fine... In an MP OS this is bad.
And right now we are only talking about Data... What if one task jumps to some code that is part of another task's context? BANG... One dead task... And exec is built around this concept!
-
HenryCase wrote:
Yes it can. Let's say you rewrite the compiler to add a "-MPAROS" compiler flag. Then whenever you try and compile programs with this flag it follows certain rules to make programs compatible with an MP kernel. The resultant code is different from compiling it normally, but crucially you wouldn't need to rewrite the application code yourself.
Let me give you an idea as to how difficult this would be. An MP version of the Amiga OS API would have to place restrictions on memory used in messages. All memory referenced in a message would have to be shared, or at least shared with the recipient. Old programs could use any memory, and might even allocate a message on the stack (which would be private). So the first step would be to re-allocate the message in shared memory.
So far so good. Unfortunately, this message could contain pointers to other memory. A compiler might know that some are pointers. However, coders have a tendency to do naughty things, and might write a pointer into a variable declared as an integer. Even if a coder does follow the rules, a pointer to an array or image, won't tell you how big that array or image is; and, if it's a dynamically allocated memory, you won't know at compile time (i.e., the compiler won't be able to do anything).
Now, to top everything off, this whole thing is recursive. The memory that is pointed to could point to other memory, which would point to other memory, etc.
This is why old apps would have to run in a sand-boxed environment. Old apps would collectively run in a shared memory space (so yes, they can interfere with one-another), and any interaction with the OS would have to go through a compatibility layer that would repackage calls to use the new API.
Hans
-
@HenryCase
I wrote a small, very simple virtual machine last weekend which you may or may not find interesting along with a simple assembler. At the very least it shows how a simple vm can be written. It's a stack machine with ~25 opcodes and no registers.
The VM (http://koft.net/static/lichen/vm.c)
assembler (http://koft.net/static/lichen/assembler.c)
example assembly program (http://koft.net/static/lichen/assembly)
It does have bugs.
-
@bloodline and @Hans_
Thank you both again for informative posts. I think you misunderstand my plan, but that's understandable as I haven't explained it fully. Think of it like API translation at the compiler level. So instead of declaring AllocMem() as a pointer, the same attributes about the memory requirements (memory size, flags) get declared in a malloc kind of way. What I aim to do is take the API calls and turn them into ones that can support MP. Does this sound more reasonable?
@koaftder
Cool. I'll check out your VM when I get time ("99 bottles of beer on the wall, 99 bottles of beer...").
-
HenryCase wrote:
@bloodline and @Hans_
Thank you both again for informative posts. I think you misunderstand my plan, but that's understandable as I haven't explained it fully.
I understand your plan perfectly!
Think of it like API translation at the compiler level. So instead of declaring AllocMem() as a pointer, the same attributes about the memory requirements (memory size, flags) get declared in a malloc kind of way. What I aim to do is take the API calls and turn them into ones that can support MP. Does this sound more reasonable?
What you haven't realised is that the function call (in this case AllocMem() ) isn't the problem. It is just an interface, the problem is how one is allowed use the API. That was defined by the behaviour of the OS and apps, in such a way as to prevent the ability to retrofit Memory Protection.
In an MP OS, one would define that the pointer returned by AllocMem() is for the use of the calling task only.
In AmigaOS the pointer returned by AllocMem() is free for any task to use, this is defined behaviour.
If you change the above definition you no longer have an AmigaOS compatible system, and everything would have to be rewritten, all the various subsystems would need to be altered in the way they work to the point that no programer would recognise the system as AmigaOS... It would probably look much more like Linux. And no Amiga Programs would compile for it anymore either...
The next problem we have is that function calls in AmigaOS are simple jumps from one executable to another... in an MP OS you have to trigger an interupt, so the machine can enter supervior mode, and then deal with the data as required.
@koaftder
Cool. I'll check out your VM when I get time ("99 bottles of beer on the wall, 99 bottles of beer...").
First I tihnk you should get to grips with how AmigaOS works... get a pen and paper and plot the interactions, once you see it written down the problems will be plain to see!
-
@HenryCase
HenryCase wrote:
@bloodline and @Hans_
Thank you both again for informative posts. I think you misunderstand my plan, but that's understandable as I haven't explained it fully. Think of it like API translation at the compiler level. So instead of declaring AllocMem() as a pointer, the same attributes about the memory requirements (memory size, flags) get declared in a malloc kind of way. What I aim to do is take the API calls and turn them into ones that can support MP. Does this sound more reasonable?
The problem is that not all messages are allocated using AllocMem() (people should be using AllocVec()/AllocVecPooled() anyway). Messages that were allocated on the stack would not work with such a scheme. You are still going to encounter every problem that we've outlined. Sooner or later you are going to encounter things that cannot be determined at compile-time.
In theory it's all possible simply because we know that humans can do it. However, you will rapidly discover that human thought-processes are way more complex than anything that you can program. Moreover, it could take a human programmer months to make the transition, including extensive testing. Expecting a compiler to do this in seconds is unrealistic.
All of what I've written above ignores the fact that a developer would structure message contents differently in a full MP environment. Pointers to structures would be used less, strings would probably be dynamically allocated instead of just using a pointer to a constant, etc.
What exactly is wrong with allowing old software to be fully-allocated in shared-memory and run in a sand-box?
Hans
-
bloodline wrote:
In an MP OS, one would define that the pointer returned by AllocMem() is for the use of the calling task only.
In AmigaOS the pointer returned by AllocMem() is free for any task to use, this is defined behaviour.
If you change the above definition you no longer have an AmigaOS compatible system, and everything would have to be rewritten, all the various subsystems would need to be altered in the way they work to the point that no programer would recognise the system as AmigaOS... It would probably look much more like Linux. And no Amiga Programs would compile for it anymore either...
It's not quite that bad. Amiga OS 4 programs can request private memory with the down-side that it's per task. There is no concept of multiple threads inside a process (or sub-processes, or whatever), so you can't perform direct sharing within a multi-threaded program. In a full MP OS, memory would be private by default, and you'd have to explicitly ask for shared memory. Henry is suggesting getting the compiler to insert the MEMF_SHARED flag (and/or whatever else woulld be used) whenever it's necessary. The problem is that finding out when it's necessary is incredibly complex.
There is no reason to completely drop message passing by reference. The only requirement would be that the message doesn't contain pointers to private memory.
The next problem we have is that function calls in AmigaOS are simple jumps from one executable to another... in an MP OS you have to trigger an interupt, so the machine can enter supervior mode, and then deal with the data as required.
Ok, now you've lost me. On all OSes, function calls to libraries are simple jumps. Sending messages between processes require an interrupt. However, that's exactly the same in Amiga OS. Amiga OS programs don't just suddenly start executing each other's code within their own task/process. If a message is sent, one task/process is put to sleep (at the interrupt), and the other is woken up by the OS so that it can respond.
Hans
-
Hans_ wrote:
bloodline wrote:
In an MP OS, one would define that the pointer returned by AllocMem() is for the use of the calling task only.
In AmigaOS the pointer returned by AllocMem() is free for any task to use, this is defined behaviour.
If you change the above definition you no longer have an AmigaOS compatible system, and everything would have to be rewritten, all the various subsystems would need to be altered in the way they work to the point that no programer would recognise the system as AmigaOS... It would probably look much more like Linux. And no Amiga Programs would compile for it anymore either...
It's not quite that bad. Amiga OS 4 programs can request private memory with the down-side that it's per task. There is no concept of multiple threads inside a process (or sub-processes, or whatever), so you can't perform direct sharing within a multi-threaded program. In a full MP OS, memory would be private by default, and you'd have to explicitly ask for shared memory. Henry is suggesting getting the compiler to insert the MEMF_SHARED flag (and/or whatever else woulld be used) whenever it's necessary. The problem is that finding out when it's necessary is incredibly complex.
There is no reason to completely drop message passing by reference. The only requirement would be that the message doesn't contain pointers to private memory.
Hans, please don't complicate things! I'm well aware of more advanced strategies used to improve performance, but Henry is having a hard enough time grasping the basics here. He needs to understand that the messages and message passing are not the problem, it's the contents that's the problem.
So I would prefer it if we could keep a examples as simple as possible.
The next problem we have is that function calls in AmigaOS are simple jumps from one executable to another... in an MP OS you have to trigger an interupt, so the machine can enter supervior mode, and then deal with the data as required.
Ok, now you've lost me. On all OSes, function calls to libraries are simple jumps.
I'm not talking about libraries, I'm talking about how you make a system cal to the Kernel. On AmigaOS Kernel calls are simple jumps the same as a library, but on a Real OS, that has a separate kernel space, you then need to use an interrupt to jump into Kernel space...
Sending messages between processes require an interrupt. However, that's exactly the same in Amiga OS. Amiga OS programs don't just suddenly start executing each other's code within their own task/process. If a message is sent, one task/process is put to sleep (at the interrupt), and the other is woken up by the OS so that it can respond.
But calling the message function does not trigger a reschedule, AFAIR... thus does not interrupt...
Anyway, my point was to show how separate address spaces actually work... Henry is missing a fundamental point of MP. It is this point we need to address.
-
bloodline wrote:
Ok, now you've lost me. On all OSes, function calls to libraries are simple jumps.
I'm not talking about libraries, I'm talking about how you make a system cal to the Kernel. On AmigaOS Kernel calls are simple jumps the same as a library, but on a Real OS, that has a separate kernel space, you then need to use an interrupt to jump into Kernel space...
Ok.
Sending messages between processes require an interrupt. However, that's exactly the same in Amiga OS. Amiga OS programs don't just suddenly start executing each other's code within their own task/process. If a message is sent, one task/process is put to sleep (at the interrupt), and the other is woken up by the OS so that it can respond.
But calling the message function does not trigger a reschedule, AFAIR... thus does not interrupt...
The other task won't receive the message until a context switch (requiring an interrupt) occurs.
Anyway, my point was to show how separate address spaces actually work... Henry is missing a fundamental point of MP. It is this point we need to address.
There's no need for separate address-spaces. Full memory-protection can be achieved without it.
Hans
-
Hans_ wrote:
bloodline wrote:
Ok, now you've lost me. On all OSes, function calls to libraries are simple jumps.
I'm not talking about libraries, I'm talking about how you make a system cal to the Kernel. On AmigaOS Kernel calls are simple jumps the same as a library, but on a Real OS, that has a separate kernel space, you then need to use an interrupt to jump into Kernel space...
Ok.
Sending messages between processes require an interrupt. However, that's exactly the same in Amiga OS. Amiga OS programs don't just suddenly start executing each other's code within their own task/process. If a message is sent, one task/process is put to sleep (at the interrupt), and the other is woken up by the OS so that it can respond.
But calling the message function does not trigger a reschedule, AFAIR... thus does not interrupt...
The other task won't receive the message until a context switch (requiring an interrupt) occurs.
Of course :-) But the actual message passing does not trigger one.. I could be wrong, here it's not really very important, we are just arguing semantics, so hopefully Henry will ignore this part of the thread.. I'll check the RKMs later and confirm.
Anyway, my point was to show how separate address spaces actually work... Henry is missing a fundamental point of MP. It is this point we need to address.
There's no need for separate address-spaces. Full memory-protection can be achieved without it.
Lets not confuse Henry anymore, please :-) We need to present the simplest example, since that gives the clearest example of why AmigaOS can't have MP...
-
Hans_ wrote:
In theory it's all possible simply because we know that humans can do it.
I think that's one of the factors that is driving me on.
bloodline wrote:
In an MP OS, one would define that the pointer returned by AllocMem() is for the use of the calling task only.
In AmigaOS the pointer returned by AllocMem() is free for any task to use, this is defined behaviour.
Okay, so Amiga software expects to be able to use pointers defined by other Amiga software. So there is almost a pool of memory pointers. Can we put these memory pointers in an OS controlled environment and control which programs we'd like to use these pointers?
In the place of the pointers in the application code we put a 'port' to the shared pointer area. When a program then wants to access the data or instruction in memory a request to the OS goes to through that port. The OS then uses the pointer itself, and the outcome of this action is stored in a 'quarantine' area in memory. The OS then tries to determine (based on past experience and/or user preference) whether that data should be given to the application.
Way I see it, if a program causes a computer to crash, we should be able to determine when it crashed, and what was the last value (data or instruction) to be passed for processing, right? If we can do this, we have options when it comes to crash protection (which is the main benefit of MP IMO), as long as we catch the value before it is passed for execution (or maybe before storage in its designated memory location).
Or maybe we don't have those options... can you give me typical low level examples of what causes a computer to crash?
Hans_ wrote:
What exactly is wrong with allowing old software to be fully-allocated in shared-memory and run in a sand-box?
I want to see the AmigaOS 3.1 API succeed as much as possible as I see it as a bridge between different Amiga-based OSs. A sandbox is okay, but it would split the community again as there would be a new 'native' API. AROS is in the unique position to bring the Amiga world together, and I don't want to see it lose that.
Hans_ wrote:
Henry is suggesting getting the compiler to insert the MEMF_SHARED flag (and/or whatever else woulld be used) whenever it's necessary. The problem is that finding out when it's necessary is incredibly complex.
Correct, I'm advocating making memory private by default. The 'finding out when sharing is neccessary' task I am trying to pass to humans.
bloodline wrote:
He needs to understand that the messages and message passing are not the problem, it's the contents that's the problem.
So I would prefer it if we could keep a examples as simple as possible.
bloodline, I appreciate your approach here. I agree it is best to keep the information you are all giving me as simple as possible, I do feel like I am slowly getting my head around the larger issues here, thank you for your patience (and everyone elses). The thing is I'm finding this discussion rather interesting, and as I'm getting more information it's throwing up more questions and possibilities in my head. Please forgive/understand my curiosity!
-
@HenryCase
Correct, I'm advocating making memory private by default. The 'finding out when sharing is neccessary' task I am trying to pass to humans.
The problem is, you dont get true memory protection. You only get partial memory protection which is already available for Amiga: Enforcer and MuForce tools.
Again everything which is passed between applications and kernel must be public memory. Using private memory in Amiga is useless because such memory can not be passed to kernel calls because you dont know what kernel does with it. You can not trust that library calls can handle private memory.
These topics have been discussed at least 15 years and yet nobody, ever, not even the best developers, could ever introduce anykind of solid idea for true memory protection. Amiga has its limits and we have to live with it.
-
HenryCase wrote:
Hans_ wrote:
In theory it's all possible simply because we know that humans can do it.
I think that's one of the factors that is driving me on.
bloodline wrote:
In an MP OS, one would define that the pointer returned by AllocMem() is for the use of the calling task only.
In AmigaOS the pointer returned by AllocMem() is free for any task to use, this is defined behaviour.
Okay, so Amiga software expects to be able to use pointers defined by other Amiga software.
Right. Data structures (which the pointers point to) can be shared between tasks and system libraries. In an MP OS you can't share the data structures.
So there is almost a pool of memory pointers.
No, memory pointers are nothing but the address of the data structure in memory. For example on a real Amiga I know that $DFF180 always points to colour register 0, so I can write to that address if I want to alter that colour register... you see a pointer is nothing but a number, a programmer could allocate some memory, and then write anywhere he chooses within that area, a pointer to which would be an arbitrary number. If a program needed to share that pointer it could use that arbitrary number in a function call or pass it to another program... the OS has no way to know that that arbitrary number is a pointer... does that make sense?
Can we put these memory pointers in an OS controlled environment and control which programs we'd like to use these pointers?
No, there is nothing special about these pointers.
The nearest thing to what you are suggesting is a sandbox...
In the place of the pointers in the application code we put a 'port' to the shared pointer area.
No, the CPU doesn't use 'ports'... it just pushes numbers around, a pointer is just a number that points to a data structure.
When a program then wants to access the data or instruction in memory a request to the OS goes to through that port. The OS then uses the pointer itself, and the outcome of this action is stored in a 'quarantine' area in memory. The OS then tries to determine (based on past experience and/or user preference) whether that data should be given to the application.
The program doesn't make a request. The program tells the CPU to load some data. No requests are made, the CPU just does it.
But you still miss the point. If you know you have a pointer then find there are ways and means to trap it, etc, its very slow... but the take home point here is that there is NO WAY TO KNOW WHAT IS A POINTER and WHAT IS JUST DATA :-)
Way I see it, if a program causes a computer to crash, we should be able to determine when it crashed, and what was the last value (data or instruction) to be passed for processing, right? If we can do this, we have options when it comes to crash protection (which is the main benefit of MP IMO), as long as we catch the value before it is passed for execution (or maybe before storage in its designated memory location).
When an Amiga program goes wrong, it can randomly write anywhere in the memory. This puts the computer in an unknown state, once the machine is in an unknown state you can't recover, unless you have a full backup of the Memory.
Memory protection means, that when the program goes wrong, it can only destroy it's own data/code... Only that task is in an unknown state.
Or maybe we don't have those options... can you give me typical low level examples of what causes a computer to crash?
If the programmer has made a mistake, then the CPU might write some data over some important operating system code.
Hans_ wrote:
What exactly is wrong with allowing old software to be fully-allocated in shared-memory and run in a sand-box?
I want to see the AmigaOS 3.1 API succeed as much as possible as I see it as a bridge between different Amiga-based OSs. A sandbox is okay, but it would split the community again as there would be a new 'native' API. AROS is in the unique position to bring the Amiga world together, and I don't want to see it lose that.
The AmigaOS 3.1 API is common to all AmigaOS clones.. so that's that...
Hans_ wrote:
Henry is suggesting getting the compiler to insert the MEMF_SHARED flag (and/or whatever else woulld be used) whenever it's necessary. The problem is that finding out when it's necessary is incredibly complex.
Correct, I'm advocating making memory private by default. The 'finding out when sharing is neccessary' task I am trying to pass to humans.
But nothing would work if memory was private by default, because MEMF_PRIVATE was never enforced on AmigaOS it was ignored... not just by app programers but the operating system designers too.
bloodline wrote:
He needs to understand that the messages and message passing are not the problem, it's the contents that's the problem.
So I would prefer it if we could keep a examples as simple as possible.
bloodline, I appreciate your approach here. I agree it is best to keep the information you are all giving me as simple as possible, I do feel like I am slowly getting my head around the larger issues here, thank you for your patience (and everyone elses). The thing is I'm finding this discussion rather interesting, and as I'm getting more information it's throwing up more questions and possibilities in my head. Please forgive/understand my curiosity!
My big concern is that you are attempting to solve a problem, without having an understanding of basic operating system design or even a simple grounding in assembler.
I think though you are missing the point, because you don't understand how a computer actually works. I think you really need to learn assembler, 68k is very easy and really fun... there are loads of good books both online and in good computer stores.
-
itix wrote:
Again everything which is passed between applications and kernel must be public memory. Using private memory in Amiga is useless because such memory can not be passed to kernel calls because you dont know what kernel does with it. You can not trust that library calls can handle private memory.
So in an MP protected OS like, for example, FreeBSD, how do instructions/data get passed between applications and kernel?
-
HenryCase wrote:
itix wrote:
Again everything which is passed between applications and kernel must be public memory. Using private memory in Amiga is useless because such memory can not be passed to kernel calls because you dont know what kernel does with it. You can not trust that library calls can handle private memory.
So in an MP protected OS like, for example, FreeBSD, how do instructions/data get passed between applications and kernel?
There are a ton of ways to do this.
Signals
Shared memory segments
IPC
TCP
With the exception of the SYS V shared memory segment thing, which is hardly ever used in my experience, all the intra process data sharing consists of passing messages from the process through the kernel and to the other process.
-
HenryCase wrote:
itix wrote:
Again everything which is passed between applications and kernel must be public memory. Using private memory in Amiga is useless because such memory can not be passed to kernel calls because you dont know what kernel does with it. You can not trust that library calls can handle private memory.
So in an MP protected OS like, for example, FreeBSD, how do instructions/data get passed between applications and kernel?
What Itix wrote may be a bit confusing because the kernel can actually read/write to whatever memory it likes (it's running in supervisor mode, not user mode), so that is not an issue. The issue is how to pass data to non-kernel OS modules. For example, much of the x-windows system on Linux runs in user-mode, and hence cannot access memory that it doesn't own. Someone else can correct me if I'm wrong, but my understanding is that Linux/BSD copies a message's content from one process' memory space to another. As a consequence, there can be no pointers to other memory in that message.
Full memory-protection means that all the data in a message (including data referenced by pointers) has to be shared with (or copied to) the other process. Thus, if an Amiga OS program passes messages to others with pointers to it's internal data, that data would have to be in shared memory (or temporarily shared); therefore, this program wouldn't be protecting its internal data. Changing such an application to use full memory-protection would essentially be a redesign of its internals, not a quick change in memory flags.
Hans
-
So in an MP protected OS like, for example, FreeBSD, how do instructions/data get passed between applications and kernel?
Memory protection and a distinction between "kernel" and "user" spaces are not dependent upon one another. You can have one without the other.
Some operating environments / operating systems / virtual machines use invalid opcodes and exception handlers to pass data between user and kernel environments or between subsystems. For example, Virtual PC uses opcode 0Fh,0C7h,0C8h and two bytes of data representing the API function to allow guest systems to communicate with the host environment.
The subsystem/sandbox approach is widely implemented in other operating systems, including consumer products, e.g. NTVDM (Virtual DOS Machine) and WoW (Windows on Windows) on 32-bit Windows NT, WoW64 on 64-bit Windows NT, FX!32 on Windows NT/Alpha, many Unixes which allow binaries from other Unixes to run natively, and Amithlon (I'm grossly oversimplifying, but it could be viewed as an Amiga subsystem on Linux).
The problem you're trying to solve is essentially the same one Microsoft solved to allow DOS and 16-bit Windows applications to run on 32-bit Windows NT. They run in either a shared or stand-alone memory space and communicate with the operating system and hardware through a transparent abstraction layer that converts software interrupts and 16-bit Windows API calls to an appropriate 32-bit Windows NT analog.
It's possible for DOS and 16-bit Windows applications to communicate natively with applications and hardware outside their sandbox by using custom modules that expose interfaces on both sides of the wall.
-
Amiga has its limits and we have to live with it.
Why should we have to live with it ?
Especially now that we have a true memory protected kernel with Quark ?
Why...
-
Wouldn't it be nice to have Aros hosted under L4 and extend from there...
-
warpdesign wrote:
Amiga has its limits and we have to live with it.
Why should we have to live with it ?
Because this is as far as the AmigaOS API can go.
Especially now that we have a true memory protected kernel with Quark ?
:-? Quark isn't AmigaOS, it's modern microkenel. It's also proprietory, if we wanted to move the Amiga platform forward I would prefer an opensource solution. L4 is a cool system, but if you really want a MicroKernel, then I would probably choose something like Darwin, which has more real world useage and is being activly developed by comercial interests.
Wouldn't it be nice to have Aros hosted under L4 and extend from there...
What is the problem with using the Linux Kernel? It's far better supported than L4, I think it was Staf who proposed a minimal Linux with AROS-Hosted on top. You've never know Linux was there and you get all of Linux's Driver support... nice idea really.
-
Back on topic...
How to improve AROS:
Instead of the wants and desired being separate, come together, form a consortium, pool the bounty money and decide in order which bounties AROS NEEDS and prioritize them. Then let programmers bid on only the top one. They declare how much money it would take them to code it... Then onto the next bounty if their is money left over... Etc... So 3 coders offer to code the top bounty, coder 1 says he'll do it for $3,000, #2 for @$2,500 and #3 for $2,750. Bids can be private, bounty is publicly assigned to #2...
-
Bloodline wrote about using the Linux kernel.
Defender of the faith about using the L4.
To Defender of the faith: What is your vision for the advantages of using L4?
-
lou_dias wrote:
Back on topic...
How to improve AROS:
Instead of the wants and desired being separate, come together, form a consortium, pool the bounty money and decide in order which bounties AROS NEEDS and prioritize them. Then let programmers bid on only the top one. They declare how much money it would take them to code it... Then onto the next bounty if their is money left over... Etc... So 3 coders offer to code the top bounty, coder 1 says he'll do it for $3,000, #2 for @$2,500 and #3 for $2,750. Bids can be private, bounty is publicly assigned to #2...
A bounty doesn't work like that. If I put $10 into a bounty it's for a spcific task, for something that I want.
Perhaps we could set up a GeneralBounty, that people can submit to, and let a "board of directors" decide where the Money goes depending upon what is considered most important. But honestly, I doubt it will be as sucessful as what we have now. There is no money in the Amiga platform anymore, what we do now is totally for fun... that includes funding bountys and developing for the various AmigaOS Clones.
-
cicero790 wrote:
Bloodline wrote about using the Linux kernel.
Defender of the faith about using the L4.
To Defender of the faith: What is your vision for the advantages of using L4?
From a geeky point of view, I'd prefer L4 as well, it has some great and unusual concepts... If we were setting up a company, that had investment and was going to produce our own hardware (i.e. of known configuration, and specific hardware) then I'd be keen to use L4.
But Linux has the advantage of software/hardware support and has been proven in the real world, plus AROS is already running quite happily on it, and integrated with it. :-)
Of course the advantage of this approach is that you can even swap out the underlying kernel, put something else there and all the AROS software would still run fine... or even lose the kernel altogether and let AROS run on it's own :-)
-
To Bloodline: I see. It's very interesting to hear great programmers talk about this, and I think the Aros will gain a substantially larger support the moment it gets modern browser support. That's all it takes. On Google you have a complete office and mail and it will be possible to use it in daily life. That will draw the old Amiga users back and the supporter base will grow bigger.
-
And with bigger supporter base, more bounty that speeds up further development.
-
bloodline wrote:
lou_dias wrote:
Back on topic...
How to improve AROS:
Instead of the wants and desired being separate, come together, form a consortium, pool the bounty money and decide in order which bounties AROS NEEDS and prioritize them. Then let programmers bid on only the top one. They declare how much money it would take them to code it... Then onto the next bounty if their is money left over... Etc... So 3 coders offer to code the top bounty, coder 1 says he'll do it for $3,000, #2 for @$2,500 and #3 for $2,750. Bids can be private, bounty is publicly assigned to #2...
A bounty doesn't work like that. If I put $10 into a bounty it's for a spcific task, for something that I want.
Perhaps we could set up a GeneralBounty, that people can submit to, and let a "board of directors" decide where the Money goes depending upon what is considered most important. But honestly, I doubt it will be as sucessful as what we have now. There is no money in the Amiga platform anymore, what we do now is totally for fun... that includes funding bountys and developing for the various AmigaOS Clones.
I know it doesn't work like that currently, but it seems people donate to what they want (niche), meanwhile what AROS "needs" remains neglected... Typically, the needs of AROS benefit everyone, not a niche set of users...
I say refund all the bounty money. Set priorities and then accept general bounty money then let the programmers with skill bid on the bounty for what they think it (and their time) is worth...
How much money would it take for you to do the Kickstart replacement? That would benefit more people (CloneA, NATAMI, MiniMig) than a port to ...even though I'm sure you'd like to port it to an ARM platform, for instance...
-
Part of the problem with a Kickstart replacement is Kickstart itself. Its internals are so well-known that it would be difficult to support a clean room design, especially on a 680x0-based platform.
Of course, you could also argue that the internals are so well-known that it's a de facto standard. The fact that Amiga hardware is no longer commercially available weighs heavily in favor of a new implementation.
Speaking of L4, if Kickstart were reimplemented, it would be very cool to see it done as a hypervisor.
@HenryCase
Don't give up. Binary refactoring (either dynamic or compiler-assisted) isn't a new concept, but it is a young one. Take a look at the literature. The traditional approach of catching exceptions and working around them will work, but it's definitely not as sexy.
Do keep complexity in mind, though. The P versus NP problem is a very real one. As noted, it doesn't take much (around 266 branch instructions or 2^266 combinations) before the number of possible outcomes is more than the approximate number of atoms in the universe (10^80). Really. All you need in that case is a quantum computer with n qubits, where n is the number of branches. ;-)
-
To cicero: I'm not sure if a browser would be the magic bullet that will bring AROS to the fore front. I still see AROS having a big problem with attracting developers to the project. The big problem, as I see it, is the inability to bootstrap AROS from inside AROS as the development environment is not complete enough to do full AROS dev inside AROS. A lot of devs will still have to cross compile from other OSes which is a pain. Not to mention that some people don't have that many free machines that they can risk knobbling.
The virtual machine AROS is a good start. This at least allows someone to not sacrifice a computer to AROS just yet. I haven't played enough with it though.
I think a good turtorial on AROS navigation, design ideas would be a good idea. AROS is unique in certain ways compared to other operating systems that someone who has never had a serious involvment with AMIGAs will be a bit frustrated and lost with AROS. Potentially another roadblock to attracting new developers to the project.
So, IMO, get that dev environment in AROS complete. From the looks of it, it is coming along but just isn't there yet. When that is done, make it clear how to develop AROS from within AROS. Even down to how to edit a file, bring up a command prompt, etc. Some basic system navigation so that new devs don't feel like retards in AROS. The more 'happy' devs you have working on AROS the more AROS will benefit. The cross compile is a bit of a killer I think.
However I found a little something over on the SDL mailing list. If a browser is wanted maybe this will be at least a bare minimum until someone has the time to port over something like FireFox.
http://www.linuxdevices.com/news/NS9883020354.html
http://www.linuxdevices.com/articles/AT5894497943.html
SDL based Web browser. AROS has an SDL port doesn't it? Or at least SDL'll probably be easier to port over than all that window manager stuff that FireFox will need. A thought.
-
To Zekaric: I was seeing this from my own point of view (what else could I do?). I was an Amiga user. It was the first in a long line of computers that had an OS, and this OS was, as it probably was for many Amiga users, almost like a second home. I left this for pc then the company crashed. I left Amiga completely but some year’s back I discovered winuae and thought it was great. And I also discovered that Amiga lived a prosperous underground life. I learned about the AROS. The first thing I did was to take out one of the older PC’s and hoped that AROS was possible to install and use for browsing. It did not work out, so I have it in VM like a goldfish. This behavior is something that I think many former Amiga owners will do with their second pc’s that stand in the basement. I do not like Linux very much. Windows is a workhorse but I would prefer to use an OS that derived from that former sweet home. I just guess that many former Amiga owners and users would feel the same. There will always be hardcore enthusiasts but I think that AROS could also attract the “lighter” former user and by that broaden the supporter base. There are a lot of former Amiga owners out there and since so much of computer time is connected to the internet I think many will return if it’s a real OS that can be use on pc for daily internet tasks. If just the browser works I think people will have patients and wait for the other bits to fall into place.
It’s just a thought.
-
Going back to the original post:
uncharted wrote:
Start off as simply a bunch of replacement files that goes on top of an OS3.1 install and keep building them up until you have replaced everything. ... Really, it's how AROS should have been done from the start.
I don't follow AROS that closely, but you mean this is not being done for the non-hardware banging code?! GCC should be able to produce both 68K and x86 files; load the 68K on a hardware Amiga in place of the regular library or whatever, test the heck out of it, and if it passes try the other version in AROS.Isn't this rather standard procedure for trying to develop identical functionality on two different platforms?
-
hardlink wrote:
Going back to the original post:
uncharted wrote:
Start off as simply a bunch of replacement files that goes on top of an OS3.1 install and keep building them up until you have replaced everything. ... Really, it's how AROS should have been done from the start.
I don't follow AROS that closely, but you mean this is not being done for the non-hardware banging code?! GCC should be able to produce both 68K and x86 files; load the 68K on a hardware Amiga in place of the regular library or whatever, test the heck out of it, and if it passes try the other version in AROS.Isn't this rather standard procedure for trying to develop identical functionality on two different platforms?
This is being done... check out AFA by Bernd
-
bloodline wrote:
:-? Quark isn't AmigaOS, it's modern microkenel. It's also proprietory, if we wanted to move the Amiga platform forward I would prefer an opensource solution. L4 is a cool system, but if you really want a MicroKernel, then I would probably choose something like Darwin, which has more real world useage and is being activly developed by comercial interests.
Darwin is not a kernal it is the OS (and XNU is not a strict microkernel)
Sorry, I don't often get the chance to be pedantic these days, have to grab every opportunity ;-)
What is the problem with using the Linux Kernel? It's far better supported than L4, I think it was Staf who proposed a minimal Linux with AROS-Hosted on top. You've never know Linux was there and you get all of Linux's Driver support... nice idea really.
Why not go the whole hog and host UAE with 68k AROS on Linux and get full compatibility? That's what I've been trying to propose.
-
uncharted wrote:
bloodline wrote:
:-? Quark isn't AmigaOS, it's modern microkenel. It's also proprietory, if we wanted to move the Amiga platform forward I would prefer an opensource solution. L4 is a cool system, but if you really want a MicroKernel, then I would probably choose something like Darwin, which has more real world useage and is being activly developed by comercial interests.
Darwin is not a kernal it is the OS (and XNU is not a strict microkernel)
Sorry, I don't often get the chance to be pedantic these days, have to grab every opportunity ;-)
You are quite right, but I was trying to think of a system that would bring with it Drivers/Hardware support... and yet was still based on a Microkernel (though I realise Mach is now really a hybrid)...
What is the problem with using the Linux Kernel? It's far better supported than L4, I think it was Staf who proposed a minimal Linux with AROS-Hosted on top. You've never know Linux was there and you get all of Linux's Driver support... nice idea really.
Why not go the whole hog and host UAE with 68k AROS on Linux and get full compatibility? That's what I've been trying to propose.
That's what I think most sensible people would want...
-
bloodline wrote:
This is being done... check out AFA by Bernd
This is not what AFA is doing. AfA only replaces a couple of libs in quite an awkward way. It is not some kind of systematic 68k port.
-
uncharted wrote:
bloodline wrote:
This is being done... check out AFA by Bernd
This is not what AFA is doing. AfA only replaces a couple of libs in quite an awkward way. It is not some kind of systematic 68k port.
It's not systematic, but it's achieving the same result... the number of bugs that have been fixed by this method is quite considerable...
-
Hi
I glanced back at my last post here and saw it could come across as rather negative. I think AROS is great and the AROS team has done a fantastic job. I Just wanted to have that said. Also, it was an issue with the first pc not AROS. The second pc I tried the AROS livecd on installed without any problem, so no shadow over AROS.
How is the current situation with the browser?
Is traveler at a standstill?
Zekaric wrote his:
However I found a little something over on the SDL mailing list. If a browser is wanted maybe this will be at least a bare minimum until someone has the time to port over something like FireFox.
http://www.linuxdevices.com/news/NS9883020354.html
http://www.linuxdevices.com/articles/AT5894497943.html
SDL based Web browser. AROS has an SDL port doesn't it? Or at least SDL'll probably be easier to port over than all that window manager stuff that FireFox will need. A thought.
End quote:
Is this the way to do it? My programming skills are not up to date, so I am of no use here.
Is it possible to get a browser going so AROS could be used as the main OS and move from pre-internet state?
-
The port of SDL to AROS is rather old but it works for some things. Also, Origyn Web Browser has been ported to AmigaOS 3 and 4 but the license isn't listed in the article. The OS 3 port might work on a hefty AROS box if Joerg is required to hand out the source.
As for Wanderer, Rob had some emotional problems to work out so he ducked out of the bounty. The bounty is valid again if you want to take it.
-
Update: It's BSD Licensed but Joerg said he would post the changes needed to make an OS4 version on the Sand-Labs Subversion server (http://www.sand-labs.org/svn/) when he's done with the latest version.
-
Oh, it was my programming skills that were insufficient, but Zekaric seems to be a dev. Maybe he could be talked into doing it. He also had some ideas for pawing the road for new devs that haven’t been deeply involved with Amiga. And all ideas that lead to more devs and more users sounds good.
AROS is different and different is good.
-
Just a little news worth mentioning:
http://www.power2people.org/bounty_018.html
M. Schulz, after his successfull SAM440EP port, is porting AROS to the Efika platform too.
-
That's terrific news, terrific work. The more platforms its on the better.
And I also really hope a talented dev picks up the browser bounty or just do it for the greater good.
I'll be waiting anxiously for ditching Windows on the home pc.
-
paolone wrote:
Just a little news worth mentioning:
http://www.power2people.org/bounty_018.html
M. Schulz, after his successfull SAM440EP port, is porting AROS to the Efika platform too.
Ahh, so it won't be long before Moto has a decent OS to run on his Lego monstrosity? ;-)
Ok then, two decent OSs... :roll: :lol:
-
I've been away all week, so I was really excited when I saw this news item:
VmwAROS to support 68K applications (http://www.amiga.org/forums/showthread.php?t=47605)
I skipped the description and went straight to the good stuff on the video page. It was interesting, but not what it said on the tin. A start, nevertheless.
Remembering this thread, my favourite part of the video page is...
Obviously, this is not the "UAE integration" many AROS users are aiming to, since it needs the original Amiga ROM and system files to work
Even if Rich had of completed the UAE bounty, it still would have needed AmigaOS system files and Kickstart Images. The first part of the Kickstart bounty is due tomorrow. It will be interesting to see if that has got anywhere.
When will you guys actually realise that a 68k port of AROS isn't some guy {bleep}ing about what he wants, but an actual requirement? It's not a good thing for an open source OS to require a closed source OS to be able to run 95%+ of the software library. :roll: Even super-duper-mega-Linux-AROS is going to need 68k port to be able to run legacy stuff (which lets face it is what the majority of the people that are going to take any notice of an Amiga-like are going to want).
I got such a kicking from the AROS community for this thread, but at every opportunity I've been proven right. The 'split' that everyone was pissing their pants about has happened anyway, and time and time again it has been shown that a 68k port is vital.
-
the split is because there was to many morphos users on the aros team, the team lost amiga coders, focus on a logo and cartoon mascot become more important than code. in the end it become difficult to stay on the right path. someone wanted aros to run on eifka, others wanted a more modern amiga lookalike os called analbus, then it all fell apart. the main goal the x86 and 68k uae integration was forgotten. as i see it now aros is pretty much dead and will not go anywhere, becouse of the split there simply aint enough coders that have any intressted anymore, sad but true.
-
Nice flamebait A1260.
-
@A1260
Aros has only lost two coders to Anubis, one of whom worked on some AROS apps a while back and the other who was planning to leave the project anyway... Now at least these guys will stay in the community...
Aros is still going strong, activity on the Dev list has rarely been higher... Get a clue before you post.
-
I don’t think this split will damage AROS, it will only future prof the Amiga as a whole. AROS don’t move forward, it kind of sprints forward. I highly anticipate the next vmwaros for there seems to be a lot of goodies for both devs and users in the next version.