This was a problem with applications (and extension) not the system itself. Even the Amiga wasn't immune to this sort of bad behavior (Amiga Basic anyone?)
Shush, we don't talk about that

Anyway, I recall quite clearly that you had to make sure you used the correct ROM images with various emulators that were 32-bit clean, since earlier ROM's were not above abusing the upper 8-bits of the address registers.
[
This isn't completely unreasonable. On the 68000 the extra overhead doesn't buy you very much, but on later machines with MMUs such a mechanism is necessary to provide proper memory protection (not that classic Mac OS took advantage of that...).
This is the same mechanism that utterly cripples floating point software using unimplemented 6888x instruction calls and invoking an exception trap. To say the overhead isn't that bad is being economical with the truth, really.
Now, if you need to go supervisor, that's fine. However, as you say, classic MacOS didn't actually bother making use of the extended control over the system that doing so would afford, so in fact, you've basically got a cripplingly slow system call that you could have implemented fine in user mode.
The biggest problems were the hackish way multitasking was implemented and that file access was handled by an application (the Finder) rather than having proper layering.
They tried quite a while to make a next-gen Mac OS that would fix the problems, but they couldn't make it work so they gave up and bought NeXT instead.
For one, command line programs are easier to script and I say this having used a number of automation tools for classic Mac OS. Certain bulk operations are much faster to do via a CLI as well. Deleting all the files that end with a certain string is trivial via a CLI with wildcards, but requires a fair amount of manual work in a GUI. This not to say that it's superior for everything, but it is handy to have around.
OSX was the best thing to happen to Macs, ever, IMO.