From using the upper 8-bits of address registers to hold data in old m68k releases,
This was a problem with applications (and extension) not the system itself. Even the Amiga wasn't immune to this sort of bad behavior (Amiga Basic anyone?)
to using exception traps for system calls,
This isn't completely unreasonable. On the 68000 the extra overhead doesn't buy you very much, but on later machines with MMUs such a mechanism is necessary to provide proper memory protection (not that classic Mac OS took advantage of that...).
The biggest problems were the hackish way multitasking was implemented and that file access was handled by an application (the Finder) rather than having proper layering.
They tried quite a while to make a next-gen Mac OS that would fix the problems, but they couldn't make it work so they gave up and bought NeXT instead.
I can't think of a single thing a cli does that a gui can't. And the gui method is much better for actual productivity, you don't have to keep listing crap out and memorising switches.
For one, command line programs are easier to script and I say this having used a number of automation tools for classic Mac OS. Certain bulk operations are much faster to do via a CLI as well. Deleting all the files that end with a certain string is trivial via a CLI with wildcards, but requires a fair amount of manual work in a GUI. This not to say that it's superior for everything, but it is handy to have around.