Madre de Dios is considered an important center of biodiversity in Peru due to its extensive Amazonian forests. However, the forests are under growing pressure due to land invasion, agricultural expansion, and gold mining. This makes support for forest management very important. This study aimed to evaluate the relationship between forest loss, land cover, land-use changes, and Brazil nut (Bertholletia excelsa Humb. & Bonpl) production in forest concessions in the Peruvian Amazon (2004–2020). Remote sensing techniques were used to classify images using the random forest algorithm, which was applied to the Landsat-5 thematic mapper, Landsat-7 enhanced thematic mapper, and Landsat-8 operational land imagery. Brazil nut production data from 2004–2020 was provided by the Regional Forest and Wildlife Service of Madre de Dios. In forest concessions, the forest area decreased continuously over the whole study period (160.11 ha/year). During the same time period, the change in forest cover in the concessions from Brazil nut to other uses was 4681 ha. At the same time, the authorization and extraction of Brazil nuts varied during the study period but did not show a downward trend. We found a significant and inverse relationship between the conversion of forest to agricultural land and Brazil nut production. However, there were insignificant relationships between forest loss, the persistence of agricultural and forest areas, and Brazil nut production. Therefore, despite the forest loss in the forest concession areas, Brazil nut production has not decreased. Production may not be affected because land pressure is higher near access roads, affecting only the areas near the roads rather than the actual areas where the Brazil nut-producing trees are located. Our results showed that nut production in non-timber forest product concessions would be negatively affected by deforestation and forest degradation, but only slightly.