Do you have anything connected to the pcmcia slot (pc card slot) on the left hand side of your amiga? If so, try removing it....alot of these RAM boards conflicted with the pcmcia port memory addressing when more than 4 mb of fast ram was used.
In addition, make sure that you delete this library (if it exists) from your libs drawer: (libs drawer location is SYS:Libs)
68040.library
To do this, type "delete libs:68040.library" in the CLI, or type it into the "Execute Command" in the workbench menu and press return.
Out of curiosity, how did you install workbench without a floppy drive? And what version is it? Did you purchase the CF card with workbench and the games already installed?