Is there something else to blame here or should I hunt down a better adapter?
I would blame ClassicWB here, but that's just 'cause I think it's rubbish. :lol:
Also, your DH0: partition is too big. I would keep it under 500MB for full compatibility with older applications. Do you really need 2GB SYS: ?
Third thought: fast ram is cheap. Add one of those generic "coprocessor+ram" cards will nearly double the speed of your machine. Again, seems like a "no brainer" to me, but your budget may say otherwise.

Fourth thought: download the full PFS 5.3 package from Aminet and format the remaining partitions on the Amiga with the pfsformat command. Do they work if you do that?