Exposure to adversity during early life is linked to lasting detrimental effects on evolutionary fitness across many taxa. However, due to the challenges of collecting longitudinal data, especially in species where one sex disperses, direct evidence from long-lived species remains relatively scarce. Here we test the effects of early life adversity on male and female longevity in a free-ranging population of rhesus macaques (Macaca mulatta) at Cayo Santiago, Puerto Rico. We leveraged six decades of data to quantify the relative importance of ten forms of early life adversity for 6,599 macaques (3,230 male, 3,369 female), with a smaller sample size (N=299) for one form of adversity (maternal social isolation) which required high-resolution behavioral data. We found that individuals who experienced more early life adversity died earlier than those who experienced less adversity. Mortality risk was highest during early life, defined as birth to four years old, suggesting acute survival effects of adversity, but heightened mortality risk was also present in macaques who survived to adulthood. Females and males were affected differently by some forms of adversity, and these differences might be driven by varying energetic demands, female philopatry, and male dispersal. By leveraging data on thousands of macaques collected over decades, our results show that the fitness consequences of early life adversity are not uniform across individuals but vary as a function of the type of adversity, timing, and social context, and thus contribute to our limited but growing understanding of the evolution of early life sensitivities in long-lived species.