The obvious thing to do if you want to remove the “problem” with mouse pointer taking CPU time, is to not use a PIO (programmed I/O) driven controller, like RapidRoad, where the CPU do all the work in reading from the devices, but instead use a controller that use DMA (direct memory access), like for example the Deneb controller (DMA with OS3, but only PIO with OS4) or - simples of all - the original Amiga mouse port.