OctaMED has its own file format but can import and export Protracker mods from it. OctaMED 6 and OctaMED SoundStudio 1.03c (the latter of which is the latest OctaMED for the Amiga) run compatibly with graphics cards. Also, SoundStudio supports up to 64 tracks, sound card support, and MIDI interface support.
Protracker 2.x is the compatible release for most mods but 3.x is around too and has slight incompatibilities making the earlier release more desirable. It does 4 voice mods and is a bag-standard tracker and bangs the hardware mercilessly. There is a hacked version that will edit 4-voice mods on AmigaOS 4.x .
As far as OctaMED and the .XM format used by FastTracker 2 for DOS and later on, MilkyTracker, they are both able to import and export .MOD files compatibly, but have a little more difficulty importing each other's files. There is a converter on the Aminet that will convert mix-mode OctaMED SoundStudio files to .XM format for use on other platforms which makes MilkyTracker more desirable than OctaMED SoundStudio for non-MIDI use.
Both OctaMED SoundStudio can be made to work on AmigaOS 4.x (using the Petunia JIT compiler and the Maestrix commodity) and MilkyTracker is a native compile for 4.x. Both would require a high-end Classic to run either of them to their fullest capability.
MorphOS can run MilkyTracker for certain. The others aren't so certain. Last I knew the Mastrix commodity didn't work on MorphOS but may have been fixed since I last looked into it.
AROS will run MilkyTracker and OctaMED SoundStudio (using JanusUAE and the Maestrix commodity I think). I don't know if there is a port of Protracker though. I sincerely doubt it. (Not that it needs it with the other two available.)
There are others available like Protrekkr on AROS and AmigaOS 4.x but that requires a HUGE amount of processor time to process the softsynth instruments. It may also be available for MorphOS but I don't have MorphOS yet so I can't verify it.