Diversified smallholder agriculture is the main human land-use affecting the western Amazon, home to the world’s richest terrestrial biota, but the scant available data to date have suggested that the biodiversity impacts of this land-use are small. Here, we present comprehensive surveys of birds and trees in primary forest and smallholder agricultural mosaics in northern Peru. These surveys reveal substantial biodiversity losses that have been overlooked by other studies. Avian biodiversity losses arise primarily from biotic homogenization across infrequently surveyed forest habitats (a loss of beta-diversity). Furthermore, tree species richness declines much more steeply than bird richness. Statistical modeling of local habitat features that allow forest-associated species to persist in the smallholder mosaic strongly suggests that our results represent a best-case scenario for Amazonian agricultural biodiversity. We conclude that previous assessments of the biodiversity value of Amazonian smallholder agriculture have been overly optimistic because they are restricted to upland habitat, thereby missing losses in beta diversity; do not evaluate trees; and/or rely on generalizations from less speciose areas of the Neotropics, where habitat specialization amongst species is less prevalent. Smallholder agriculture will likely expand in western Amazonia due to infrastructure development, and it must be seen as a serious threat to the region’s biodiversity.