Possibly because the "C64 was cool." Or maybe because the 6502 & decendants didn't go on to become the core of mainstream computers.
There was 16-bit variant which had two operation modes, emulation mode and native mode (like 80286 had real mode and protected mode) and yet another variant was used by Nintendo in SNES.
Yes I see you only mentioned mainstream computers ;-)
Yes, a compiler can take care of the bank switching for you. Plus, the 80486 also had a mode allowing full 32-bit RAM access without switching.
80386 had that already. 386 was actually quite decent chip unlike its predecessor, 80286.
But to me, as a developer, it does not matter if there is bank switching involved or if CPU is using slower base+index segment memory model. Only user experience is important.