Recent studies point to a rapid increase in small-scale deforestation in Amazonia. Where people live along the rivers of the basin, customary shifting cultivation creates a zone of secondary forest, orchards and crop fields around communities in what was once was old-growth terra firme forest. Visible from satellite imagery as a narrow but extensive band of forest disturbance along rivers, this zone is often considered as having been deforested. In this paper we assess forest disturbance and the dynamics of secondary forests around 275 communities along a 725 km transect on the Napo and Amazon rivers in the Peruvian Amazon. We used high-resolution satellite imagery to define the ‘working area’ around each community, based on the spatial distribution of forest/field patches and the visible boundary between old-growth and secondary forests. Land cover change was assessed between ca. 1989 and 2015 using CLASliteTM image classification. Statistical analyses using community and household-level data from the Peruvian Amazon Rural Livelihoods and Poverty (PARLAP) Project identified the predictors of the extent of forest disturbance and the dynamics of secondary forests around communities. Although shifting cultivation is the primary driver of old-growth forest loss, we find that secondary forest cover which replaces old-growth forests is stable through time, and that both the area and rate of expansion into old-growth forests are modest when compared to forest conversion in Peru for colonization and plantation development. Our findings challenge the notion that smallholder agriculture along rivers is an important threat to terra firme forests in Amazonia and point to the importance of protecting forests on community lands from loggers, colonists and other outsiders.