If you do not have a graphics card, the OS will use chip memory for storing images. So if you use a large resolution with lots of colours (arguably impossible under AGA), a colourful wallpaper, PNG icons, and a snazzy VisualPrefs theme, you may find the amount of chip memory reduced considerably.
8000 bytes remaining is rather excessive however - that sounds more like the processor memory is not being used at all for any tasks.
I would recommend using FBlit, which should improve the speed of graphics redraws, while also forcing any graphics related tasks to use fast memory (where possible).
Cheers,
James
x