@Jose
You are gonna hate me for this but I have a suggestion...
First of all, try splitting your large function into 2 or 3 helper functions, it makes it a hell of a lot easier to read and maintain :-)
Anyway, assuming I understood it correctly, you are working on the assumption that every pixel other than the start of a line would be a HAM pixel. What you might consider doing is finding some code for quantitative colour reduction and finding the best 64 colours from the source data. You can then use these as a base palette for your calculation and try to find (within each pixel row) if it is better to use a palette colour or a HAM pixel.
I'm sure most good RGB ->HAM conversions would do something like this anyway.
As for your compiler problem, I don't generally use SAS-C, so I am not sure I can help there :-/
