Maxtransfer depends on your host adapter. You might want to try smaller values to make sure nothing goes south. Lower values will increase overhead, thus might be slower. To high values may lead to crashes/read/write errors. As a non-expert for CS I'd say 0x00FFFE00 should be fine though. The best values should be in the manual, really.
The mask value defines were DMA buffers get allocated. It would be reasonable to exclude chip and Z II RAM as they're dog slow but the masking scheme more or less only allows you to limit the upper boundary of buffer space. However, as long as there's 32 bit RAM available it'll get used since it's got a higher priority.
If your problems happen before you run out of accelerator memory, I'd guess it's a RAM defect. If things happen when you run out of accelerator mem I'd double check the onboard RAM, just remove it for testing.
