I've never heard any discussion (going back to hanging around the dev team in '86) regarding a cartridge port. However, since it was originally designed to be a computer disguised as a game box it is quite possible there could have been a stealth add-on to handle carts.
Carts were associated with game machines and toy computers. At the price the Amiga was selling at, being compared to a game machine was a very bad thing at the time.
The infamous bust of the video game market in the '80s seems to have ended the use of cartridges on consumer machines until game machines underwent a revival in the '90s. Amiga came after the bust and, one might say, pushed the revival of the stand-alone box which could provide better graphics (and games) than the PCs of the day.
