You know what! Somehow, this was related to screenmode. I changed to 1920x1080@50Hz and everything is now fine!
1920x1080@60 was listed as "Monitor Native Mode" but with that selected, I got no sound and amiberry was running very slowly (30fps and the audio buffer light was flashing).
Very strange as 60Hz is definitely the "correct" mode for my TV (Sony Bravia 4k thing, couple of years old).