In my experience on an Amiga, your processor is going to be the largest impact in speed. RAM would be second, depending on how you expand it, since there's no virtual memory on Amigas. Hard drive access time would be next in line (ergo the popularity of using compact flash drives).
On a modern PC, RAM is more critical to performance since the more you have, the less swapping to virtual memory occurs. CPU would be next in line followed by hard drive speed (for when swapping does occur).
[edit]Holy cow you guys are fast! There were no responses when I started writing...[/edit]
