There are a lot of things that can slow down when hardware acceleration is enabled (ironic as that sounds).
There are certain rules to using HW acceleration. You need to lock the hardware for as short as possible but also as infrequently as possible (as the lock incurs an overhead). It can be tricky to get an ideal balance so it is common for people to just lock the hardware for a whole frame and only unlock it to refresh the display. Due to this, user settings can also influence things too, when the hardware is locked, multitasking and interrupts may also be disabled on some configurations. This can really chew the system up.
A utility called WarpPrefs allows you to easily access all of the relevant envarc settings.