That's the problem with custom chips, you can possibly design a wideo chip that will come close to the Nvidia chips of today, let alone those of 2010, wo12 etc. A custom chip is in you machine forever, if I want to upgrade to the latest Nvidia card it takes 2 minutes to open the case, pop out one card and pop in a new one. Plus Nvidia has thousands of employees working on these cards. There's just no way a few hundred Amiga fans can outdo Nvidia, Custom chips just create headache because some joker thinks they can write directly to them and then you can't upgrade because your software breaks.
There's really two roads here, a modern Amiga inspired by Classic Amigas or a reimplimented retro machine with a tweek here and there. There's a fundamental choice. You need to choice, abandon state of the art or abandon classic except as UAE.
There is no way to make classic modern, we are at the same fork in the road Apple was at in 1997. I'm not saying we need to make the same choice, Apple pretty much owns "state of the art" and the competition is hard, maybe a retro games box has a better chance, retro-gaming with the some retro-computing on the side may be a good niche for Amiga. The Amiga has been state of the art in a decade and a half.
The sad thing is, no matter what the choice the community will grow smaller, and neither way offers a guarantee of success.