IIRC, at least on the 68060, the floating point unit is already faster for a lot of common arithmetic operations than fixed point would be, especially for multiplication which 3D transformation naturally uses a lot.
On paper, the 68060 integer MULS/MULU is 2 cycles while fp FMUL is 3 cycles. However, speed isn't just measured in the shortest timings. The integer MULS/MULU is pOEP only not allowing for parallel operations. The FMUL can operate in parallel with integer instructions (a characteristic of 68k coprocessors predating the 68060). A compiler with good instruction scheduling can usually do the most in parallel with mixed integer and fp. Sometimes no integer instructions during floating point instruction execution is possible though. Converting from fp to int and int to fp is 3 cycles. If the 3D engine is setup to use floating point then it's usually not worth trying to convert to fixed point but a properly designed fixed point 3D engine probably is faster even on the 68060. The lack of integer 32x32=64 bit makes fixed point more difficult and slower on the 68060 though.
P.S. I haven't seen a 68k compiler do proper instruction scheduling with FPU instructions. Frank Wille and I were talking about the possibility of adding to vbcc as well as improving the FPU support in general. I don't know if the poor code is the limiting factor or the cache sizes with complex 3D scenes. If the limitation is the code quality and not the caches, it may be possible to run Quake like engines at a lot better frame rate with an improved 68k compiler. Quake 3 would probably need GL support and a fast 68060 though
.