Alzheimer’s disease (AD) includes a long period of presymptomatic brain changes. Different risk factors are associated with AD development, including having a family history of AD (FHAD). The Braak scheme suggests that tau pathology, in synergy with amyloid-beta (Aβ), spreads along structural connections in AD, eventually leading to atrophy. Studying the pathways in which atrophy spreads early on, as well as the factors underpinning this pathway, is crucial for improving diagnostic accuracy and early interventions. However, the pattern of atrophy progression in people with a FHAD and the biological factors associated with this progression remain unclear. Here we used structural MRI from three databases (ADNI, PREVENT-AD and Montreal Adult Lifespan Study) to map the atrophy progression in FHAD and AD and assess the constraining effects of structural connectivity on atrophy progression. Cross-sectional and longitudinal data up to 4 years were used to perform atrophy progression analysis in FHAD and AD compared to controls. Positron emission tomography (PET) radiotracers were also used to quantify the distribution of tau and Aβ proteins at baseline. We first derived cortical atrophy progression maps using deformation-based morphometry from 153 FHAD, 156 AD, and 116 controls with similar age, education, and sex at baseline. We next examined the spatial relationship between atrophy progression and spatial patterns of tau and Aβ deposition, structural connectivity, and neurotransmitter receptor and transporter distributions. Our results show that there were similar patterns of atrophy progression in FHAD and AD, notably in the cingulate, temporal and parietal cortices, with more widespread and severe atrophy in AD. Both tau and Aβ pathology tended to accumulate in regions that were structurally connected in FHAD and AD. The pattern of atrophy and its progression also aligned with existing structural connectivity in FHAD. In AD, our findings suggest that atrophy progression results from propagating pathology that occurred much earlier, on an intact connectome. Moreover, a relationship was found between the serotonin 5-HT6 receptors spatial distribution and atrophy progression in AD, supporting an important role of these receptors in neurodegeneration. The current study demonstrates that regions showing atrophy progression in FHAD and AD present with specific connectivity and cellular characteristics, uncovering certain of the mechanisms involved in preclinical and clinical neurodegeneration.