Motivation: It is increasingly common for international development actors to work in difficult and fragile settings, yet much of our understanding of accountability through social and political action comes from more stable settings. As an increasing percentage of the world's population live in places characterized by fragility, it is essential to understand the context for accountability in these settings. Purpose: This article proposes a framework based on three domains of contestation: natural resources, identity, and legitimacy. Based on this framework, it unpacks the constraints towards progress on accountability in difficult settings, and the challenges in creating an environment for citizen claim-making. Methods and approach: The article draws on research from a five-year Action for Empowerment and Accountability (A4EA) programme which focused on examining social and political action in Mozambique, Myanmar, Nigeria, and Pakistan. Research papers from the programme were analysed together with a close reading of existing literature to decipher the common traits hampering citizen action and state response in difficult settings. Findings: The core argument is that a weak state and the prevalence of non-state groups vying for power are fundamental to difficult contexts in three different domains: natural resources, legitimacy, and identity, in which underlying contestation generates conflict, violence, and fragility. The three domains are set within two structural factors: historical legacies and social norms. The article traces how contestation in these domains within structural factors influences the behaviour of powerholders as well as the outlook of citizens who could make accountability claims. Policy implications: The article concludes by suggesting that lessons for development actors attempting to promote accountability in these settings are: to "work with the grain", e.g. recognize the context and work with it; explore possibilities for change at the local level, e.g. where alliance building and networking might be more feasible; and look for the small wins-because in a context fraught with difficulties, small achievements such as overcoming fear should be counted as a success.