Purpose: Neuronal activity during face matching shows co-activation of the fusiform gyrus (FG) and areas along the ventral visual network. To elucidate the mechanisms related to the facial discrimination deficits in Alzheimer’s disease (AD), the study evaluates the relationships between β-amyloid (Aβ) load and gray matter (GM) atrophy within the ventral visual network.Methods: Comprehensive cognitive assessments and GM volumetry using 3-dimentional T1-weighted images and AV-45 positron emission tomography (PET) were studied in 44 patients with AD. We used AV-45 PET to measure regional Aβ to analyze the correlations between the regional neocortical AV-45 retention and atrophy in patients with AD.Results: FG volume was positively correlated with the para-hippocampus (β = 0.565, P < 0.001), posterior cingulate cortex (PCC; β = 0.402, P < 0.001), and hippocampus volumes (β = 0.209, P = 0.044). After carefully confounded all possible factors simultaneously, the hippocampus standardized uptake value (SUV) ratio was independently associated with FG volume (β = −0.151, P = 0.017). Furthermore, volumes of the hippocampus (r = 0.473, P = 0.003), para-hippocampus (r = 0.515, P = 0.001), and FG (r = 0.383, P = 0.018) were associated with Benton’s facial recognition test (BFRT).Conclusions: In conclusion, our study indicated that amyloid burden within the hippocampus might contribute to FG cortical hub GM atrophy. While the face matching task scores were related to the FG, hippocampus, and para-hippocampus volumes, concordant changes of the aforementioned three structures suggested the importance of the three ventral visual network hubs in AD.