Antiretroviral treatment requires high levels of adherence to be effective. This qualitative study explores the reasons for poor adherence among 22 purposively selected poor urban participants in South Africa. Over a 4-month period in 2009, we prospectively investigated experiences of HIV diagnosis and treatment, adherence, and withdrawal from treatment. Patients with no stable food sources faced significant barriers in adhering to treatment regimens and staying sufficiently healthy to search for, obtain or retain a job. The narratives also identify the influence on adherence of self-esteem and social support, vulnerability generated by ill health, gendered conflict, social inequities amplified by HIV, and exhaustion due to the social burden of the epidemic. Multi-dimensional, inter-sectoral programs that tackle the social determinants of health, such as food insecurity, poverty, gendered inequities, and treatment adherence are more likely to be successful, than single interventions to support adherence.