Adding libraries will have no effect on system performance, as they're only loaded to memory on demand.
Datatypes are slightly different: The datatype descriptors are all loaded by AddDataTypes at system startup. When a unknown file is opened with datatype system, the descriptors are used to identify the file. This identification process can potentially get slower if tons of datatypes are installed. In practice this is no problem however, as there are limited number of usable datatypes.
