There is growing interest in technologies that allow older adults to socialise across geographic boundaries. An emerging technology in this space is social virtual reality (VR). In this paper we report on a series of participatory design workshops that involved extended in-depth collaboration with 22 older adults (aged 70-81), that aimed to understand their views on the types of social VR experiences that they saw as being of value to older adults. This process culminated in a reminiscence-based social VR concept. Our study identifies: participants' ideas about the types of social VR experiences they found appealing; the potential for social VR as a powerful reminiscence tool; how social VR might be used as a tool to challenge ageing stereotypes and promote healthy ageing. Reflecting on the design process, we discuss how the diverse participant groups and the complexities involved in mediating between designers, the technical team, and participants could inform future design work.