Should be possible.
A year ago, in a hurry, I quickly fired up my Rev5 A500 with AExplorer file executed on stripped-down version of Workbench 1.3.2 (v34.28). I managed to copy few complete disk images, with no errors.
IIRC, that setup involved hacked serial port driver (BaudBandit.device) and reduced number of bitplanes (Add44K) to get some CPU cycles available.
It seems you're experiencing problems with transmission errors - stock 68000 is too slow for your baudrate. Small files will get through, but a bit larger than a buffer size might become an issue.