First of all, Quark was never viable on the Amiga,
Although I'll not say you are not correct in the fact that Quark maybe made the mistake of not advancing their software, or basing it too heavily on one technology tied to an OS, the main reason why Quark lost out on the Page Layout market is due to the Adobe's decision to release Creative Suite, after that it was all over because you basically got InDesign for free, which negated all risk for publishers/designers to give it a try as Photoshop/Illustrator/Acrobat were basically de facto anyway. And this was when InDesign was only really just pulling ahead feature/usability wise from Quark, but it was enough to pretty much convince everyone to switch.
But going away from that for a minute,
early on, Quark would have worked on an A2000 no problem, in '92-'94 there was a ton of magazines etc.. being laid out on monochrome/grayscale screened Macs. And having seen Pagestream 3, I found that to be quite similar in feel, albeit minus the Mac interface differences.
Add to the fact that due to the graphics market was the main provider of sales it naturally drove the hardware to tailor itself to it. In '86-89 I don't think a lot of what we take for granted as given qualities for the Mac platform for graphics was as locked down or inevitably to follow as may be suggested when looking at historical timelines. What I said was that Commodore should have gone after that segment of the market, which would have entailed responding to it accordingly. The point made above about Shapeshifter literally out performing Mac's on equivalent or lesser hardware only confirms that it was very possible. Even my lowly AGA A1200 030 back in the day ran Shapeshifter pretty nicely, certainly usable, not for high end stuff photo manipulation of course, but then the equivalent Mac wouldn't have been chosen either, but many design/layout projects could be undertaken quite satisfactorily.
It's not always about the high-end, yes it's great bragging rights, but you get the usability sorted along with a good price/performance ratio, then if you can get critical mass, you're in. Amiga was in many ways most of what the Mac promised to deliver but failed for many years.
In some ways, all Commodore would have needed to do was to show you could continue working while copying files from a Syquest drive, the amount of time wasted up till OS X, OS 9 was terrible in that aspect, and when time/deadlines = money, it makes a pretty convincing argument. That is, if it had been made in the first place of course.