I don't think they planned to port it back, or they would have done 68k first.
When AROS was started, AmigaOS had "official" support... Whatever that actually turned out to be... It wasn't the intention of AROS to get in the way that.
From what I can tell it's mostly the graphics HID that is the problem, as well as sub optimal compilers.
Compilers tend to make the code a bit larger than hand coded ASM (naturally), and you correct about the graphics, no one has actually written an "Amiga Driver" yet, it still just does all graphics operations as readPixel/writePixel (the emergency fallback gfx mode of AROS), fix that and you will have sorted the major bottleneck.