I think the install process was designed with 68K in mind, and, unfortunately, OS4 chokes on a lot of 68K stuff. Try installing manually - the only thing qtools does is extract the contents of the .pak files, which you should also be able to do with plain ol' UnArc.
Hopefully my install log can help you figure out manual installation.
Create New Directory: "ENV:Quake2"
Create New Directory: "ENVARC:Quake2"
Create Text File: "ENV:Quake2/InstallationPath"
================== Start of textfile ==================
A1200-4:Quake2=================== End of textfile ===================
Create Text File: "ENVARC:Quake2/InstallationPath"
================== Start of textfile ==================
A1200-4:Quake2=================== End of textfile ===================
Copy file "Q2Ami:libs/wizard.library" to "A1200-4:Quake2".
Copy file "Q2Ami:libs/WesSupport.library" to "A1200-4:Quake2".
Run Program: "assign libs: Q2Ami:libs add"
Run Program: "t:qtools x "Quake2:install/data/baseq2/pak0.pak" -dd "A1200-4:Quake2/baseq2""
Run Program: "t:qtools x "Quake2:install/data/baseq2/pak1.pak" -dd "A1200-4:Quake2/baseq2""
Run Program: "t:qtools x "Quake2:install/data/baseq2/pak2.pak" -dd "A1200-4:Quake2/baseq2""
Copy contents of drawer "Quake2:install/data/baseq2/Players" to "A1200-4:Quake2/baseq2/players".
Run Program: "t:hUnpack -archive "gui.hpa" -dest "A1200-4:Quake2""
Run Program: "t:hUnpack -archive "bin68k.hpa" -dest "A1200-4:Quake2""
>Do you want to install the Movies ?
Ask Choice: User selected "Yes".
Create New Directory: "A1200-4:Quake2/baseq2/video"
Copy contents of drawer "Quake2:install/data/baseq2/video" to "A1200-4:Quake2/baseq2/video".
>Please choose which Mod's you want to install (if any)
Ask Options: User selected following options...
Threewave CTF
Create New Directory: "A1200-4:Quake2/ctf"
Run Program: "t:hUnpack -archive "Mods:CtfPPC.hpa" -dest "A1200-4:Quake2/ctf""
Run Program: "t:hUnpack -archive "Mods:Ctf68k.hpa" -dest "A1200-4:Quake2/ctf""
Remember to get the update from Hyperion's website (assuming they ever put it back online) and to replace wizard.library with the OS4 native version.
If there's an OS4 native build of Quake 2, though, you should definitely use that.