Scotland is well known for having the worst health in Western Europe, with the country's premature mortality mainly driven by suicide and substance misuse, rather than physical illness. These problems only emerged relatively recently, and a similar profile of premature mortality, sometimes called "diseases of despair", can be seen in other societies at different times. But what is "despair" in this context, and how might it exert its effects? Studies of the impact of Adverse Childhood Experiences (ACEs) on adult health consistently reveal a similar profile of morbidity, which is thought to relate to the effects of unremitting, unsupported, "toxic" stress. As models of childhood adversity expand to include a wider range of causes and powerful mitigating factors, there is a need to understand why some life events and circumstances are especially harmful. This paper argues that socioeconomic factors, childhood adversity, attachment, resilience and "toxic" stress are all consistent with a broader concept of human potential: Nussbaum's "capabilities approach" to human development. Incorporating our understanding of health and wellbeing in the broader frame of capabilities strengthens our understanding of adversity-related harm, and might also point towards new ways of repairing the social and individual damage they cause.