A great deal depends upon which Amiga you're talking about!
68000 (16-bit data bus) Amigas (1000, 500, 600) perform noticeably better with TRUE (still 16-bit) fast ram. Adding a 32-bit processor (68020, 68030, 68040,68060) alone is a mere modest boost. A 32-bit accelerator with 32-bit fast ram gives a large boost in performance.
Everything depends on which Amiga and how far you ultimately intend to expand it.