How exactly does the Amiga RTG system work?

[Piru]

@gazsp

how exactly does the system know how to use the card?

User selects graphics card screen mode for desktop and/or applications.

Does graphics.library just get patched so that all OS calls go to the cards library / drivers instead of using the native chipset?

graphics.library gets patched indeed, but only to make it possible to use the standard functions on chunky 8bit and truecolour screens. Typically original chipset modes remain functional (both CGX and Picasso96).

[Piru]

@gazsp

Is it just the files in Devs/Monitors that inform the system of stuff like where the screen memory is located, what format the data is in (i.e. planar, chunky) etc. ?

Well, not really, as the OS itself has no knowlege of such things. It only knows about planar stuff. However, CybergraphX or Picasso96 patch the system so that if gfx card mode is used, correct things are displayed. All such details as the framebuffer address and size are internal to the graphics card system software.

I just don't really understand how it all gets set up initially in software so that the system knows to stop using (reading and writing to / from) planar Chip RAM, and start using the memory / GPU on the graphics card.

There is a good reason you don't: It doesn't. Only OS friendly apps using proper OS calls to do the rendering work. Any HW banging apps will always use the native graphics output, or display only partially on gfx card output (missing parts in GUI for example).