Speed rankings of Next Gen hardware

[spirantho]

Those timing graphs are completely irrelevant. They were made by Piru (a MorphOS developer), and do not compare like with like (the different versions were made on different people's systems with a different version of the software, probably compiled by a different compiler) - they're not comparing one system with another, they're comparing a truckload of variables, making the comparison useless for anything except propaganda.

It can be difficult to say which is "faster". On some tasks, a Sam 460ex will be better than a Pegasos II (for instance tasks which require better infrastructure - the 460ex uses PCI Express for starters). On others, a slower G4 Peg II will slaughter a 1.1GHz 460ex if the task is Altivec enabled.

I believe the order for most uses of AmigaOS 4 is:
AmigaOne X1000
Pegasos II G4
AmigaOne XE G4
Sam 460 1.1GHz
Sam 440 Flex and EP (faster CPU, 800MHz)
Pegasos II G3 600MHz
Sam 440 Flex and EP (Slower CPU, <733MHz)
AmigaOne XE G3
Micro-A1
CS-PPC
Blizzard PPC

Of course there are variations - I believe the fastest BlizzardPPC outruns the fastest CyberstormPPC, for instance. Then of course some apps use the faster bus of the Sam compared to the AmigaOne XE, that sort of thing.

Generally speaking, though, the Peg-II is a good board to have if you don't mind being limited to pre RadeonHD graphics cards and a PCI/AGP bus. If you want to run more modern graphics cards you need a Sam460ex or best of all the X1000.

[spirantho]

Those timing tests are dead-on accurate and reflect all other tests made
by big and professional companies I read ... Those tests can be used to
give you a very clear picture of what you're buying ...

If you showed that timing graph to a real benchmarking forum they'd just laugh at you.

1) What version of the tools benchmarked was being used? If not the same version, then no comparison can be made.
2) What compiler was used for each system? If not the exact same version of the same compiler, then no comparison can be made.
3) What RAM did each computer have - did each platform have the best RAM available? If one platform was using sub-optimal RAM, and another was using fastest available, then no comparison can be made.
4) Do the tasks involve HD access? If so, what HDD was used? Was one using an SSD and another an HDD? Did one have DMA available and another not? Was one SATA, and another PATA?
5) Were each of the benchmarks carried out on the latest versions of the OS on a clean installation? If not, no comparison can be made.
6) Were all background tasks halted before the benchmark was executed? If not, no comparison can be made.
7) Was the network disconnected from each machine? If not, no comparison can be made.
Does one of the systems support SSD instructions and one not? If so, then the benchmark is meaningless except in the area of SSD apps, and should not be used arbitrarily.

Benchmarking between systems isn't just a matter of running a program with the same parameters on two systems and timing the difference. You have to be certain that ALL other parameters are equal.

I remember the thread that spawned those benchmarks. Piru had gone through the forums and found results from other people, regardless of their system. I also know that his benchmarks were using an older version of the tool under AmigaOS 4.

These "benchmarks" were just numbers trawled from forums, and have no meaning whatsoever, as they only show 10% of the full picture.

[spirantho]

If you a attempting a scientific test of a microprocessor running a specific bit of code then the above might be valid, sometimes.  But the result will be completely meaningless because that's not what people buy.

It's still meaningless. If you want I can benchmark a Core i7 and have it produce results slower than an Amiga 500.  That's why it's meaningless, you need to know all the facts and there has to be constraints.

If a benchmark isn't the same across machines, then it has to be transparent, and the differences must be documented.

It's kind of like running quake benchmarks. One person runs it at 1600x1200 with a software renderer, the other plays it with Quake and runs it at 320x200.

Which is the better computer? without all the facts, you can't say. Ergo the benchmark is useless.

It is not some small part of the system being compared but the entire thing.  It's how fast can system X produce a result compared to system Y.

Which only makes that benchmark valid for that operation using that software, and shouldn't be used in the computing equivalent of "my dad can beat up your Dad" arguments.  It tells you nothing because it doesn't tell you where the bottleneck is.
You could run the same task now and have a wildly different result, therefore you're testing the software (which can get changed) and applying it to the system. That is not good benchmarking! At the very least it should be made clear the EXACT system being benchmarked, the software used, the compiler used and the compilation options. For all we know he could have used numbers from a debug version!

Consider the Top500 supercomputer list.  It tests systems where *everything* is different.  You are not only allowed to modify the source code but you are expected to!

I suspect those will more tightly regulated. I severely doubt the  Top500 is worked out by trawling the forums for numbers and not even knowing what changes were made and what the systems were, which is how that graph came about

[spirantho]

Ehm...

Ok...

We are very scientific here, arent we?

Don't see anything wrong with that

I mean, good that you are critical. But denying benchmarks and providing your own chart based on personal feelings does not compute

I'm not "denying" benchmarks, I'm just saying that they're not providing the whole picture, which they're not. Heck, we don't even know if Altivec was enabled for both platforms, and if so in what Altivec was used.

My chart is an impression from my own usage of hardware and from what other people have said. But I'm not passing it off as fact, that's the difference - it's just an (educated) impression.

What's most annoying, though, is that the OP was quite specific about what he wanted - an OS4 speed comparison - and yet the same old MorphOS benchmarks get wheeled out of retirement. Who cares how fast a G5 Mac is when you want to run OS4?