As far as for the first ST models compared to A500, no, there was no competition. The ST didn't have a blitter and only 16 colours on screen simultaneously. At that time TOS wasn't multitasking and playing modules when using other programs (it could be done using something called "accessories", I think it's kind of like commodities) slowed down the machine.
Later on, Atari modified their ST models to include a blitter and they also released the MEGA STe with a 16MHz 68000. This machine was of course faster than Amiga on vector games and such. I don't think it sold very well though, since it was a "pizza-box" machine, kind of semi-professional, and thus quite expensive.
The ST/TT models didn't have HAM modes, but they could produce something similar using clever programming tricks. There was even a paint program using this graphics mode.
Before getting my A1200, I was kind of interested in getting a Falcon. I never did, because I didn't know anyone who owned an Atari and I didn't know anything about the OS, so I got a second-hand A1200 instead.
However, the Falcon had some impressive specs. Just like the A1200 it had a built-in hard drive interface. AFAIK it also featured a SCSI connector, the famous MIDI port, the DSP, and newer graphics circuits. I don't beleive The 16MHz 68030 was that much faster than the 14MHz 020 in the A1200, but the Falcon came fitted with 32-bit memory as a standard (C= should have done this with the A1200... grr...)
The graphic modes on the Falcon were much like AGA AFAIK. The truecolor mode couldn't be used in all resolutions like HAM8 on AGA, but in return it was a REAL truecolour mode with no fringing.
I've never owned an Atari, so this is just written from memory of what I've read in magazines, brochures and on the net.
Atari conquered the music market thanks to the MIDI interface, I don't think the DSP made that much of a difference (perhaps it can be used for C2P, but then again I don't know if the Falcon needs C2P). Amiga conquered the video market so I think the impact on the differnet professional sectors was about equal.
There is a living Atari community: Lately, a 68060 turbo was released, there are several replacement desktops/OSes and new software is being developed. I get the feeling that the remaining Atari community is not as big as the Amiga community, but that might just be because I'm an Amigan :-)
I hope to get hold of a used Falcon one day just to see what it was all about, but currently they seem to be very rare, at least in Sweden.
I noticed a post from an Atari-guy a few days ago, perhaps he could step in and sort things out? I'm sure there are errors in my Atari ramblings :-)