Nice to see your advice/ideas/concerns.
First, I never planned to find a sensor that output Amiga compatible signals. I can program Atmel AVR's so that's not a problem. I just have to learn how to read the data from the sensor and then convert it to Amiga format. If fact, I have a couple of spare ATtiny2313 with should be useable in a prototype. I probably could find a smaller one that is more production friendly.
I bought a cheap mouse online and I will gut it and see if I can use the sensor, lens and LED. What worries me a bit is the space inside the mouse case. But with the ball out of the way there will hopefully be enough space to mount the sensor/lens without having to damage the mouse case.
A new PCB has to be designed. I guess I can use a protype board to begin with. I haven't designed a PCB for many years but I guess it's like riding a bike...
Yes, I could get a proper optical mouse and a signal converter if I wanted, but I like the challenge. I became and engineer because I like to build and develop stuff. The end use is secondary.