ABSTRACT. In this study we examine poverty in local fisheries using a social-ecological resilience lens. In assessing why "fishery may rhyme with poverty", Christophe Béné suggests a typology of impoverishment processes, which includes economic exclusion, social marginalization, class exploitation, and political disempowerment as key mechanisms that accelerate poverty. We extend his analysis by exploring these four mechanisms further and by intertwining them with processes of environmental change and degradation. Our goal is to understand poverty in local fisheries as a process rooted in social and institutional factors as influenced by ecological dynamics. We argue that understanding poverty will require a focus on the social-ecological system (SES) as a whole, and addressing poverty will mean rebuilding not only collapsed stocks but the entire social-ecological system, including restoring relationships between resources and people. Information from two cases, the Chilika Lagoon on the Bay of Bengal in India, and the Paraty region on the southeastern coast of Brazil, is used to understand how fishery social-ecological systems come under pressure from drivers at multiple levels, resulting in a range of impacts and pushing the system to a breaking point or collapse. We analyze elements of what it takes for the whole system to break down or collapse and push fishers into poverty and marginalization. The Chilika SES has already broken down, and the Paraty SES is under pressure from multiple drivers of change. The two cases help contrast key dynamics in the social, cultural, economic, political, and environmental spheres, for lessons on system collapse and recovery. Rebuilding fisheries may be examined as a process of building and strengthening resilience. The challenge is to make the fishery social-ecological system more resilient, with more flexibility and options, not only within fishing activities but also within a range of other sectors.