char hexOut[len*2+1];
It looks like a compiler bug. One of the compilers I have to support didn't work with variable length arrays until very recently, I still avoid them.
I never used vbcc, I always stuck to gcc 2.95 and sas/c. But vbcc is still supported, so you may be able to get it fixed. It's a pity there isn't a modern gcc or clang build targeting 68k.
"Please support vbcc by contacting the authors if you find any bugs or problems. Supporting eight different architectures makes testing extremely time consuming, so this release is probably not free of bugs.
For problems with the compiler core contact Dr. Volker Barthelmann (vbemail), and for Amiga/Atari-specific problems, including assembler, linker, startup-codes and linker-libraries, contact Frank Wille (fwemail)."