HAM is supposed to be used on HSV values not RGB where the 'fringing' is less jarring due to the more subtle differences when HSV values are being manipulated on an image but Jay Miner decided it was better than nothing with RGB after seeing it in action on a Flight Simulator system.
The flight simulator is where he got the idea for HAM from, it wasn't an Amiga flight simulator. But when the Amiga output went from HSV to RGB as they pivoted from a games console to a desktop computer then they were going to take it out, but decided it would be extra work and they didn't have time to reuse the space for anything else anyway.
Photo realistic images was not even the goal, it was for fast horizontal line filling. Instead of setting each pixel to blue/red/green etc you could set the pixels where the color changes. I don't know if the HSV chipset worked better, but there doesn't seem to be a way of setting up memory with a NO-OP value. So it pretty much fails to be useful for the original intended use.
