Custom chipset card would probably only make sense in a 68k system since on a PPC you have to emulate anyway.
A custom chipset could be made by using some programmable logic chips and some VHDL code so the lack of chips would not be a problem. Could be a cool project but probably pointless.
I would recommend writing a simple custom chip to GFX board mapper instead by using the CPU's MMU feature to allow running some porly written apps that don't promote very well. For games that make full use of the hardware there are UAE. (Not the 'United Arabe Emirates')
