Our memories of episodic experiences are represented and organized by contexts. Events that share overlapping contextual features such as spatio-temporal proximity are more likely to be organized together in our memories. But how do we represent and remember other people? The predominant view in social cognition is that people are represented as a bundle of attributes such as personality traits. However, in real-world contexts we most often learn about others by observing or participating in social interactions, which inform our understanding of interpersonal-relationships. In this study, we explore how people spontaneously represent and remember others after watching a character-driven television show. We find that memory for individual characters is organized around interpersonal relationships rather than person- or situation-specific attributes (i.e., actions, traits, social groups, or locations). Moreover, even the subtle order in which participants recalled the names of each character can be predicted by graphs of that character’s social network. These findings suggest that the way we naturally represent and remember people is largely driven by their connections with others. We believe our work provides new avenues of inquiry that move beyond person-centric approaches in social cognition and a novel perspective on contextual features for memory research.