Agriculture as we understand it did not occur during the ice age. In fact the standard definition for defining when the Neolithic begins is evidence of farming.
Humans tended to forage as a group, and they would roam as far away from the protection of their cave as was necessary. However, they would defecate near their cave. And, as with other animals, ours contained the seeds of what we had eaten as well as a ready source of fertilizer.
Humans tended to choose the same caves for habitation for the same reason - proximity to water and food. This meant that when the next group of humans stayed in the area they would be more likely to have a ready supply of nuts and fruits. The system was self-perpetuating, and likely improved our odds of survival the longer we lived in caves.