Tau pathology has been associated with neuronal loss at autopsy, but the temporal evolution of tau pathology and atrophy remains unclear. Here, we investigate the association between cross-sectional AV-1451-PET as a marker of tau pathology and cortical thickness cross-sectionally. We also investigated retrospective rates of cortical thinning over the three years preceding the AV-1451 scan in a clinically normal cohort of 103 older adults from the Harvard Aging Brain Study. Tau measurements were Geometric Transfer Matrix partial volume corrected standardized uptake value ratios (SUVRs) with a cerebellar gray reference region. Thirty-four FreeSurfer-defined cortical regions of interest (ROIs) were used for both thickness and AV-1451 in each hemisphere, with seven additional volumetric ROIs. We examined “local” relationships between AV-1451 and cortical thickness in the same ROI, as well as inferior temporal AV-1451 and all thickness ROIs. All models included baseline age and sex, both interacting with time in retrospective longitudinal models, as covariates. Cross-sectional models controlled for the number of days between the two scans. Cross-sectional local comparisons revealed significant associations between elevated AV-1451 and thinner cortical ROIs predominantly in temporal regions, while analyses associating inferior temporal AV-1451 with all cortical ROIs showed a widespread pattern of significant relationships, which was strongest in temporal and parietal cortices. In our retrospective longitudinal analyses, we saw significant relationships in temporal and parietal regions. Significant local relationships were seen in right superior temporal, middle temporal, temporal pole, and fusiform, as well as the left cuneus and banks of the left superior temporal sulcus. Significant relationships between inferior temporal AV-1451 and faster thinning were observed in right temporal regions (middle temporal and fusiform) and bilateral parahippocampal cortices. We observed significant negative relationships between local and inferior temporal AV-1451 signal and both cross-sectional cortical thickness and rates of thinning in lateral and medial temporal regions. This is an important early step toward elucidating the relationship between tau pathology and retrospective longitudinal atrophy in aging and preclinical AD.