What I am looking for is an explanation of how datatypes work. For example what happens when the user clicks on a file?
You are mixing datatypes with DefIcons. When a user clicks on a file, datatypes are not involved at all. Up to 3.1 the file to be clicked on is not even identified. All files without icons are assumed to be programs.
Only with NewIcons or OS 3.5 the DefIcons program was introduced. It lets Workbench examine the file to determine a better icon for it. But this is not done with datatypes.
Datatypes need to be used explicitely. Multiview for example uses datatypes to determine the file type and to load/display/play back the file.
How does the system know which datatype is responsable for which file?
You don't need to know this. It's internally to datatypes.library. You just use the functions of datatypes.library in your program, for example NewDTObject and GetDTAttrs.
Probably datatypes.library iterates through all available data types. The descriptors in devs/datatypes contain the information how to identify a file type. Usually the file is opened only once by datatypes.library and a small buffer is used for comparisons which contains the first few bytes of the file (64 bytes IIRC).
Most datatype descritors are held very simple, for example "compare the first 12 bytes with the pattern FORM????ILBM". The comparison is done by datatypes.library, there is no foreign code involved.
Only very few desctiptors actually contain code. In this code everything can be done, though. For example the code can close and reopen the file and read it entirely if needed. But this should not be done because identification needs to be very fast.
Once the file type is identified and the application wants to load the file, the class library from classes/datatypes is called. This library contains the code to decode the file contents and to make it available to datatypes.library.
I am not an experianced amiga user and therefore do not know where to look for this information.
Everything you need is contained on the Amiga Developer CD. This CD is a real treasure of information.
First you should learn about BOOPSI. If you understand BOOPSI, it's easy to understand datatypes, ReAction, MUI, ...