I have tested some more, and tried with a 512 byte buffer instead of 2048 bytes. Started the transfer, and 13 minutes later it has completed a third of the whole transfer with adfverify disabled. It stops, giving me an error saying "There has been a conflict accessing the Amiga Explorer resources" so it seems like it drops the connection down the line.
Booted my A600 with CF hdd, also with only 1 MB ram and 68000 CPU, and started the transfer of the same file, this time with adfverify enabled and 4096 bytes buffer, and the transfer completes succesfully in around 9 minutes.
Still a long time to wait for getting a single disk done, so still very slow, but at least useable. I guess an A500 simply isn't up for the task.
-edit-
Sorry, just checked an had adfverify disabled on the A600 as well.