Current trends in gerontology conceptualize Virtual Reality (VR) as a tool for rehabilitation, lauding its potential for cognitive rehabilitation or as an intervention to reduce cognitive function decline. However, we must take a critical stance and identify not just the potential positive impact, but also how things may go wrong without appropriate guidelines, and the need for careful design around the interaction affordances of the technology. We conducted co-discovery and co-design workshops involving expert stakeholders and older adults (N = 25) over a period of 6 months, involving practical activities including user personas and focus groups to understand the complexities of loneliness and identify possible solutions with VR. Based on our findings we focus our argument on two key factors in the conceptualization of loneliness: spaces, and activities which may take place within said spaces. We present our reconceptualization of VR as a tool for group activities instead of passive consumption of content and make suggestions to the community for reducing feelings of loneliness with VR.