I totally agree that without some form of standardisation the community will fracture.
I think the majority of us should be able to see this after what has happened on the Amiga and in this thread.
From a CPU perspective, I see absolutely no point adding or changing any instructions - I'm focussing on functional and timing accuracy for the 68000, then performance for the 68020+.
Personally, if you are going to mess around with the architecture sufficiently to force a compiler modification, you might as well recompile to something else entirely. ARM or MIPs spring to mind.
There is probably a minimal benefit for a 68k CPU enhancement for the FPGA Arcade or any other hardware focused on emulation accuracy right now. There are customers who want higher performance CPU cores and will want to run software compiled for higher performance cores like Apollo/Phoenix though. You can offer a retro compatible core without enhancements and a high performance core with enhancements provided the standard was not too difficult. ColdFire compatibility should be appealing for Atari Firebee users, program sizes could be reduced by 5%-15% to better fit the limited storage space and some FPGA Arcades may even sell for embedded purposes (the Raspberry Pi has sold many units for embedded purposes although it is cheaper and smaller). MIPS programs would be approaching twice the size of 68k+CF programs and they need that much more caches too. Thumb 2 is competive in code density with the 68k but a 68k+CF would be better and can have better single core and memory performance. Most of the CPU evaluation and testing would happen on more performance oriented hardware. Custom chipset enhancements are obviously higher priority.
For the chipset I have already made a few obvious improvements, such as extending all the DMA address register widths. There is not particularly controversial as there is space to do this in the memory map.
I don't expect software to use it, but if this sort of enhancement could be documented and agreed on, it becomes a possibility.
Documenting and making public the changes would be a good start. The custom chips are not my strong point but I would think RTG/chunky custom chip enhancements and maybe some improvement in the audio department would be wanted by most FPGA projects.
Matthey: I would contact Linux/m68k developers, gcc/glibc developers and NetBSD developers and ask if they would be interested in participating.
There are developers assigned to the 68k GCC backend but what they do is usually minimal maintenance. It wouldn't hurt to try inviting them to participate. As far as BSD/Linux 68k developers, there aren't very many active 68k developers and most need an MMU (ThoR and Frank Wille could give some insight if they were interested).
The standard is the existing hardware (processor+chipset) and the rest is done by the OS (perhaps with special optimized libraries)
Is 68020+AGA enough for everyone? These are well defined and it's not a bad standard but is that enough forever? If we want more, then we should create new standards.