Beaten to it, but all the same...
I don't know whether it ought to be different on an A4000, but this is how I might try on an A1200.
Partition (if required) and format the small hard disk - you could do this with nothing else attached, maybe - and make it bootable and install Workbench 3.0/3.1 on it. Nice and clean and simple. It's only 2-3 megabytes for 3.0. You could do this off your Workbench floppy disks. Method can be the same as is used later on (*!). Update whatever 3.0/3.1 needs in order to cope with the big hard disk. (That might mean getting 3.1 ROM.)
Turn the machine off. Fit the big disk as well as the small one. Check any jumpers to see that master and slave are correct.
Turn the machine on. Partition and format the big hard disk. Make at least one part bootable.
(*!) Open the Workbench, choose Show All Files, Select All, and drag the whole lot into the bootable part of the big disk.
Turn the machine off. Remove the small disk. Check jumpers.
Turn the machine on. See if it worked.
If so, turn the machine off. Fit the CD, checking jumpers.
Turn the machine on. Now is the time to make 3.0/3.1 into 3.9 (provided that you have better than 3.0 ROMs). Install all your stuff into the big hard drive from the CD.
Send me the 2.1GB hard drive. I only have a 700 odd MB hard drive (Workbench 3.0 with a fair bit of 3.1 system software in there), so I could use it. (Edit: by all means wipe or format it first.)