@Thorham
Hit the hardware is issue of speed vs comparability / flexibility, for the user it means it will limit there hardware choice..
It depends on what you want to do, for most part you should avoid banging the hardware, but if you are interested in making a OCS game for Amiga500, it makes the most sense to take advantage of every CPU cycle you can get, and try to avoid the overhead of the software API.
Writing a game for a modern graphic card, and using CIAA/CIAB timers, will exclude many people from using your software (AmigaONE-*/Sam4*0), there for its not a good idea.
Poking around in $DFF00A and $DFF00C, to get Joystick and mouse, will limit your software to only to work with 9 pin joysticks, and mouse, and will not allow your software to support USB mouse and Joysticks.
Many people have USB upgrades to there classic Amiga (Algor/Subway), and NG users don't have 9 pin bus mouse at all.
Other things people do that is annoying is that they assume they know BytesPerRow, this forces the game or program to only run in one screen resolution, and fixed mode ID's that can't be changed.
For examples if you write game/program to use Paula sound chips, then it might be faster on a MC68020, where don't have lot of CPU power, but you will not allow your game/program to use a modern sound cards like Prelude 1200 or any other sound card.
Sticking to graphic.library will make your game or program run on any hardware but, it will most likely run slower then if you wrote directly to memory, instead of using graphic draw functions, and so on.