How do we mentally organize our memories of life events? Two episodes may be connected because they share a similar location, time period, activity, spatial environment, or social and emotional content. However, we lack an understanding of how each of these dimensions contributes to the perceived similarity of two life memories. We addressed this question with a data-driven approach, eliciting pairs of real-life memories from participants. Participants annotated the social, purposive, spatial, temporal, and emotional characteristics of their memories. We found that the overall similarity of memories was influenced by all of these factors, but to very different extents. Emotional features were the most consistent single predictor of overall memory similarity. Memories with different emotional tone were reliably perceived to be dissimilar, even when they occurred at similar times and places and involved similar people; conversely, memories with a shared emotional tone were perceived as similar even when they occurred at different times and places, and involved different people. A predictive model explained over half of the variance in memory similarity, using only information about (i) the emotional properties of events and (ii) the primary action or purpose of events. Emotional features may make an outsized contribution to event similarity because they provide compact summaries of an event’s goals and self-related outcomes, which are critical information for future planning and decision making. Thus, in order to understand and improve real-world memory function, we must account for the strong influence of emotional and purposive information on memory organization and memory search.SignificanceOur brains enable us to understand and act within the present, informed by previous, related life experience. But how are our life experiences organized so that one event can be related to another? Theories have suggested that we use spatiotemporal, social, causal, purposive, and emotional dimensions to inter-relate our memories; however, these organizing principles are usually studied using impersonal laboratory stimuli. Here, we mapped and modeled the connections between people’s own annotated life memories. We found that life events are linked by a variety of factors, but are predominantly connected in memory by their primary activity and emotional character. This highlights a need for theories of memory organization and retrieval to better account for the role of high-level actions and emotions.