@Kenny,
I don't think it's so nutty. It doesn't take much to topple civilisations. If there were an event (impact, volcanism, other) that cut agricultural output by a few percent (especially back when folks lived a little closert to the edge), the food riots and privation would wreak havoc and town folk would have to go out and establish and work their own fields to get enough to eat. It wouldn't be a sudden wiped out by big rock kind of end, but it would spawn resource wars and famine which, over time, would topple a lot of civilizations.
It's much easier to be civilized when your needs are met.