Given the current demands for biodiversity conservation and climate mitigation, understanding the dynamics of fragmented forests in one of the most threatened regions of the Atlantic Forest is essential for supporting small and large-scale forest restoration initiatives. We investigated the patterns of forest loss, recovery, and stability from 1985 to 2020. We classified and quantified forest cover using temporal and spatial descriptors and related them to their causal agents. We hypothesised that there is a spontaneous natural regeneration process in abandoned areas, protected areas, and near the hydrographic network, but anthropogenic land uses negatively affect this process. Consequently, we observed that despite an increase in forest cover over the years, there was a loss of medium and large fragments of old-growth forests due to agriculture and urban infrastructure expansion, especially in flat areas and near watercourses. The increase was mostly comprised of small young forests in abandoned and protected areas, especially near watercourses. Forest stability was explained by proximity to municipality administrative units. This research elucidates the dynamics of forest patches in a severely threatened Atlantic Forest region. It revealed the replacement of large and medium mature forest fragments with small fragments of young forest. Thus, our results reinforce the need for monitoring forest stability, avoiding forest loss, and incentivising forest gain for biodiversity and ecosystem service conservation.