I had an A2091 with a PP&S 040 as well and it was damn slow. Took 3 minutes or more to load a 3.5 minute stereo song into Samplitude for editing.
I eventually got a GuruROM but it didn't help much.
On the other hand, I never knew anything about changing memory priority. How then heck do you even do that?
I had 32MB 32bit RAM on the PP&S 040, 4 on the GVP card and 4 on the CV643D... that system was really sluggish initially with a stock 3.1 install, but after many days and month spent tweaking the setup, it was quite fast... I seem to recall the issue being that the ram was not autoconfig, so a lot of libraries ended up in 16BIT fast ram.
I also recall it getting about 3.5MB/sec SCSI with the Guru ROM and 400K/sec on the Hydra Ethernet. ShapeShifter really flew on it... I ended up having a HD crash and did not have the system partition backed up and it took a few days to figure everything out again. I seem to remember getting the Guru Rom because I was struggling to get the CD-ROM to work but being really happy with speed increase.
Yes, 3.5MB sec is not fast in modern terms, but in Amiga terms its not bad. My A1200 with FastATA gets 5.1 for example. Its a heck of a lot better than the stock IDE controller in the newer machines.