Good idea to remove the battery before it leaks. Looks like it might be a bit furry which is the first signs it is about to leak. The Amiga will work fine without the battery.
If you are considering getting a hard drive, rather than screw around with ancient hard drives and a non-booting controller why not just get a Buddha? The Zorro II IDE controller by Individual Computers. The A2000 is the one computer that it is actually worth getting one. They are only
CD $62. You will then be able to use IDE drives upto 128GBytes and IDE optical drives (CD-RW/DVD-RW) etc. You could even get an
IDE to Compact Flash adapter. Install it into one of the front bays. It would make it VERY easy to transfer files between the PC and the Amiga!!
NOTE: I've heard that the newest Kingston CF cards are a bit fussy with Amiga IDE controllers, while I think this refers to A600/A1200/A4000 internal IDE controllers, best to be safe and avoid this brand. SanDisk and Leisurestore are known to work well.