It looks like a nice purchase!
For capacitors there are lists online of the types you need. I did a quick look on Google and got multiple results. You can also buy capacitor kits which have all the right ones from suppliers like Amigakit.
A gotek will replace the floppy drive, so if you want a hard disk solution look at an IDE to Compact Flash adapter. Again you can get preconfigured ones from Amigakit and others; otherwise you need to install the operating system from scratch. Also not all compact flash cards work with Amigas.
Last recommendation is to get a network interface or a pcmcia to compact flash adapter kit. Either of these will be a massive help to get files and programs onto you Amiga.
I can't help regarding the midi interface.
