I'm also curious as to why there is no problem booting into the Workbench floppy disk as it does not have a 68040.library or a 68060.library in the Libs drawer?
Most likely because the programs on the Workbench disks haven't gotten any code aimed which would tricker either 040 or 060, so therefore doesn't crash even if you run it without the CPU libraries.
Heck, you might even be able to boot the install disk without CPU libraries installed as well, but the safe way is to have the CPU libraries installed.
Why bother with the Install disk anyway? All that is needed on there is HDToolbox, so you can easily make a copy of your Workbench disk and have HDToolbox on that if you delete some unneeded files.