<b><i>Objectives:</i></b> A growing body of research has documented associations between social relationships and cognitive function, while findings are less clear regarding specific aspects of social relationships that are relevant to change in cognitive function. Furthermore, it is unclear whether associations differ at the between-person and within-person levels. <b><i>Method:</i></b> The present study used 8-year longitudinal data from the Health and Retirement Study (HRS) to examine the role of structural (partnered/married, number of social network partners, and contact frequency) as well as functional (support, strain, and loneliness) aspects of social relationships for episodic memory at the between-person and within-person levels. Analyses are based on up to 3 waves of data from 19,297 participants (mean age at baseline = 66 years, SD = 10, range = 50–104; 58% women). Control variables include age at baseline, gender, education, functional health, and depressive symptoms. <b><i>Results:</i></b> Findings showed that at the between-person level, most structural and functional aspects were related to levels of memory performance, with participants with higher numbers of social network members, more frequent contact, and more positive experiences outperforming others. An exception was a higher number of family (child or relative) relationships. At the within-person level, on occasions where participants had a higher number of close family relationships than usual, had more social contact than usual, and felt less lonely than usual, they also showed higher than usual episodic memory performance. Finally, negative effects of social strain and loneliness on episodic memory performance at the between-person level were moderated by social network size, indicating that effects were more negative among individuals with larger social networks. <b><i>Discussion:</i></b> Both structural and functional aspects of social relationships contribute to between-person differences in levels and fluctuations of episodic memory performance. Ups and downs of relationships to relatives, social contact, and feelings of loneliness contribute to ups and downs of episodic memory. Potential mechanisms underlying these associations are discussed.