Amiga.org
Amiga computer related discussion => Amiga Software Issues and Discussion => Topic started by: ptek on December 24, 2006, 07:49:49 PM
-
A few moments ago I just had this (maybe) wacky idea :
Unlike projects as SETI@home and other distributed computing systems that takes use of computers across networks (Internet included, of course) in order to create a gigantic computing system for a specific task, why not to create a client/server system where any CPU intensive program (like a mpeg2 encoder/DVD authoring) could be individually allowed to share processing resources across the network ?
The second option would be more demanding to come true : to change an existent OS (AROS, for example) or virtual machines like QEMU or UAE (considering UAE as a virtual machine) to make use of the distributed computing.
The third option could be in the form of a library (or DLL on the PC winblows) that could be called to send out processing tasks to the network and wait for the results)
This is not necessary Amiga specific... It just a thought about distributing computing. But if applyied to PC or Amiga, it could allow the conversion of a DivX to DVD a snap, for example.
Does this already exists the way I just described ?
Could this be done ? I think so, even if it would demand a inteligent managment of task, in order to avoid unnecessary "freezes" (the tasks of window dragging/moving and some common things doesn't need to be distributed and shouldn't)
This should be developed as an open source project of course to avoid any suspicion on privacy of the data exchanged.
-
That still doesn't sound too much different than what S@H does, which is just a wider area version of what render farms do. Except, that your idea seems like it would increase inefficiency and cause bandwidth sucking, because what it sounds like you're saying is not to send all data needed for a task to each computer and have it work on it alone, but to use each computer like a separate CPU. To decrease the inefficiency, you'd have to send large large chunks of data instead of swapping bits back and forth. But, then you'd be back at the S@H/renderfarm idea.
I poopoo your idea. :-)
-
ptek wrote:
A few moments ago I just had this (maybe) wacky idea :
Unlike projects as SETI@home and other distributed computing systems that takes use of computers across networks (Internet included, of course) in order to create a gigantic computing system for a specific task, why not to create a client/server system where any CPU intensive program (like a mpeg2 encoder/DVD authoring) could be individually allowed to share processing resources across the network ?
The second option would be more demanding to come true : to change an existent OS (AROS, for example) or virtual machines like QEMU or UAE (considering UAE as a virtual machine) to make use of the distributed computing.
The third option could be in the form of a library (or DLL on the PC winblows) that could be called to send out processing tasks to the network and wait for the results)
This is not necessary Amiga specific... It just a thought about distributing computing. But if applyied to PC or Amiga, it could allow the conversion of a DivX to DVD a snap, for example.
Does this already exists the way I just described ?
Could this be done ? I think so, even if it would demand a inteligent managment of task, in order to avoid unnecessary "freezes" (the tasks of window dragging/moving and some common things doesn't need to be distributed and shouldn't)
This should be developed as an open source project of course to avoid any suspicion on privacy of the data exchanged.
Screw you. You can have my CPU cycles when you pry them from my cold, dead fingers.
-
@ptek
Does this already exists the way I just described?
What a great idea, and just imagine a Beowulf cluster (http://www.beowulf.org/) of these!!!
[/slashdot]
;-)
edit - fixed quote tag
-
The problem with Asymmetric MultiProcessing is load-balancing. Getting them to finish a portion of a task at the same time makes the job difficult because there is no way to know how all of the capabilities of dissimilar computers will be able to complete.
-
Cracking idea, but one big flaw well two ok three na loads.
1. Latency....as someone has suggested work packages go off and don't come back, great for file sharing, awful for rendering, encoding etc. The more real-time it gets the more the latency will delay it all.
2. Priority....if we go by the low priorty stuff being simple stuff like data packets and high priority being audio video e.t.c, then the likelyhood is your packet data packets will be nak'ed added 1. Slowly does it.
3. The most important intensive tasks that a computer processes often need nanoseconds per clock cycle, e.g Graphics has its own dedicated processor e.t.c, to harness that kind of power would be a little beyond calling the kernel.
4. As another poster has gestured, this kind of work requires local processing, multicore and farmed on a 1gbit+ lan. For high definition medical ography, e.g Radiology, MRI,CT,XRay e.t.c which does use this kind of approach, the minimum spec is 1gbit between stations using dicom.
Cheers
Shaz
-
Screw you. You can have my CPU cycles when you pry them from my cold, dead fingers.
It seems that you didn't catch my idea. I refered the existence of a CPU usage manager, so the idea was not to use 100% of the other's CPU but only slices of it. Otherwise, would be unfair.
BTW : I find the "screw you" expression an offensive one. Respect the others if you want respect for you.
-
Interesting stuff, the Beowulf ...
-
To decrease the inefficiency, you'd have to send large large chunks of data instead of swapping bits back and forth
Not all task should be distributed, even if this system would be integrated on the OS... That's why the need of a CPU manager to decide the need to handle the task locally or not.
I think the approach of selecting only the programs which we wanted to take advantage of this "farm" would be best, since there would be an unavoidable delay between task requests and their completion by the other remote computers. Of course the data to be processed would have to be sent is some quantities and not bit by bit, along with the "instructions" of what to do with it for the remote(s) computer(s) selected to handle it. There should be a correct management of their resources, so only close to idle computers would take the work, but never taking 100% of their CPU ! Everyone who decided to join the "farm" should benefit, like the ones who are at P2P SW like bittorrent.
Besides, I thought this working multi-platform. So this could be a good use for intel/AMD many Ghz power machines to do in their idle times. Classic Amigas with theirs motorola CPUs would suffer a bit. Maybe AROS running on x86 would be a better scenario.
The difference from SETI@home is that any program/task may be distributed. SETI only handles SETI task :)
This are just thought of someone (me) that doesn't have any pratical or theoretical knowledge of distributed computing, so I named the idea like "wacky". But, who knows, maybe some of what I described here could work.
-
This is not my field at all, but isn't this exactly how xgrid (http://www.apple.com/server/macosx/features/xgrid.html) works?
"With Xgrid, scientists, animators and digital content creators can run a single job across multiple computers at once — without ever rewriting a single line of code."
Sounds like a multi-purpose CPU cluster to me?
-
First, there's effeciency problems. When working with things like MPEG decoding, the resulting data stream tends to be huge and that clogs network bandwitdh. Getting data in and out of a CPU is often more of a problem than making the CPU itself fast enough. Networks are thousands of times worse!
Second, distributed computing only works well with massivlely parallel computations. Many tasks are not, which is why even multithreaded applications for multiple cores are slow to take advantage of the new dual and quad-core CPUs. It's very impractical on many levels, and that's just why supercomputers are still in demand, instead of having long been replaced by larged networked farms of cheap computers.
Third, the actual benefit is not really that huge. It looks good on paper, but real-world results are a different story (the PS3 has this problem, big time).
ptek: It seems that you didn't catch my idea. I refered the existence of a CPU usage manager, so the idea was not to use 100% of the other's CPU but only slices of it. Otherwise, would be unfair.
That's pretty much how they work now, using WAIT instructions as appropriate to reduce waste. Power distribution tends to be a problem with modern chips, though, and adding transistors to redistribute power just adds peak-power usage. If you open up a CPU case these days, you're going to see big plates of copper covering the surface of the chip to distribute power, rather than an interesting grid of wires. Promotional photos of microchips rarely reflect the chips actually packaged in real equipment.
Task scheduling is a much debated problem with modern kernel design. How does the OS tell the difference between a lazy loop and an intensive tight loop? The application is really responsible for telling the OS how much CPU time it needs, and you can't expect programmers to cooperate like that. Taking advantage of an external idle CPU, or trying to shy away from peak-performance computing is very difficult. There *are* reasons why it's not done.
-
What you are describing is part of MacOS X, it's called XGrid... LAtency of the internet make it unsuitable for WAN usage, but if you have a gigabit network at home it's a treat.
Apple's Logic Pro 7.2 uses this system to allow all Macs on your Lan to help out processing the audio signal. I currently use 3 Macs on a Gigabit network just for audio processing (Two MacBook Pro's and a PowerMac G5). It works brilliantly and all in realtime!
-Edit- Opps beaten to it :-)
-
bloodline wrote:
What you are describing is part of MacOS X, it's called XGrid...
And QNX (QNet), and OpenMOSIX (http://openmosix.sourceforge.net/), and what DragonFly BSD is working towards, and...
It could be interesting to stuff some sort of generic sandboxed VM (the JVM, Parrot, etc) into inetd highly-niced (mm, jargon), but you'd still have to find something high-latency, unreliable nodes are good for. It *would* take the onus off the user to remember to update S@H/F@H clients manually or through native mechanisms, but you'd also lose the candy that attracts people to some of those projects ("ooh, pretty screensaver").
Edit: This probably reads more like English with the 'to' put into the paragraph above, now that I notice.
-
There is an idea to create on demand cluster from PowerPC G5 (Apple XServe) by Omneta (September 2005).
I do not know if this project is real or just a hoax.
Apparently it failed
Perhaps in the future independent smart robots can access remote massive AI database/CPU power.
http://www.omnetaplc.com
From MacObserver:
"A new supercomputing cluster built from Apple Xserves will be launched in Europe. eWeek reported Wednesday that Omneta had signed a Memorandum of Understanding (similar to a Letter of Intent) with Apple to provide Xserves and Xserve RAID units for the cluster, and another MoU with Interoute to handle networking services for the project.
Omneta intends to offer the computing power from the cluster to customers on a one-time or monthly basis, and if demand is there, the cluster will be expanded to several "secret, ultra secure, bombproof centres" across Europe, according to a statement issued by Livo Technologies. Livo Technologies will be providing the system of logging into the network.
eWeek reported that Omneta Chief Technical Officer Tony Clark attributed Apple for providing "significant support" for the project, and that the company may even help market the service once it is launched. Mr. Clark also told the magazine that Apple could supply up to 500 Xserve units per day if needed.
The business model for the startup is aimed at corporate clients in a market with many political borders. For instance, it will be possible to make sure that data being processed on the networked cluster is contained within geographical borders, as might be required by local governments or corporations.
Ironically, it's the power of the G5 processor that brought Omneta to Apple. Mr. Clark chose the company after Steve Jobs said publicly that the PowerPC G5 was the most cost-efficient, highest-bandwidth processor in the data center."