>The data words from the modulator channel are written into the corresponding registers
>of the modulated channel each time the period register of the modulator channel times out."
I read that as meaning the following:
Suppose channel 0 is the modulator and channel 1 the modulated channel.
Channel 0 has playback period of its own, just as it would if it were playing a sample. Instead of feeding the data to the DAC, here the data is fed to the period register of the other channel and that this happens at the basic playback period of the modulator channel.
The "each time the period register of the modulator channel times out" part just seems to be a needlessly long way of confirming this.
Therefore, if your modulator channel has a period equivalent to 5kHz playback, it will update the period register of the modulated channel 5000 times a second. Note that this isn't yet the basic requirement for FM synthesis as it could be writing the same value to the modulated channels period register 5000 times a second, therefore changing nothing.
One thing which seems to be the case here is that building an FM synth would be quite difficult. Rather than simple frequecny modulation, you would be performing period modulation (period is proportional to the inverse of the frequency) meaning your modulator waveform would have to be appropriately encoded.
The bit about data words rather than bytes just means that the 16-bit elements fetched by the modulator channel now represent a register field rather than a pair of 8-bit samples. The period register has a 16 bit resolution, so you can envisage your modulator waveform as an 8-bit sine wave, converted into a 16-bit period value.
This of course means that you need to start with some fixed wafevorms that you can encode in this fashion.
It looks difficult overall. Difficult, perhaps impractical, but not impossible.