It can make a difference if you use FastFileySystem on very large disk partitions and they decide to validate (which can lock up an Amiga for hours, so don't do it.)
The thing with the Amiga is you can always run out of memory by multitasking too much.
I haven't had much experience of big Amiga memory TBH, I would reckon Lightwave, ProPage 3 or AdPro/MorphPlus would be the real hogs.
More is always better but you can still run out.
I would think you would be OK with 64.
Maybe happier with 128 and you are working with silly size bitmaps from a scanner.