In this paper we examine evidence from rock shelters in the Northern Province that points towards a complex sequence of interaction between foragers and farmers. Farmers underwent radical shifts in social complexity that had a range of implications for change in the identity, status and viability of hunter-gatherers in the region. Based on a composite excavated and rock art sequence, we argue that rock shelters were places of social power, of which hunter-gatherers gradually lost control in the face of farmer appropriation, particularly in the second half of the 1st millennium AD. We argue that this appropriation stemmed, in part, from farmers appealing to the ambiguous power of 'first peoples' as a resource in the regulation of their own social needs. Farmers consequently expropriated rock shelters by overwriting, adding to and subtracting from, and recycling hunter-gatherer deposits and images by imposing their own set of marks. There is some indication that the manipulation by farmers of rock shelters as ritual arenas retained and extended some specific hunter-gatherer belief. This continuity is slight, however, and, consequently, we suggest that major rock shelters were features that were inherently powerful resources. As most ethnographic observations of farmer response to foragers have shown, farmers were ambiguous in their categorisation of foragers. They assimilated their powerful places at a general level but, at the same time, expunged and rewrote the meaning and power of place in their own terms. We argue that, in the Northern Province, the farmer assimilation of forager places led to major forager settlement and social change. Based on the sequence we present, we conclude that interaction cannot be reduced to either/or responses underpinned by terms such as association, on the one hand, and assimilation on the other. The process and the outcomes were far more complex.