Objective: Typical aging is associated with gradual cognitive decline and changes in brain structure. The observation that cognitive performance in mesial temporal lobe epilepsy (TLE) patients diverges from controls early in life with subsequent decline running in parallel would suggest an initial insult but does not support accelerated decline secondary to seizures. Whether TLE patients demonstrate similar trajectories of age-related gray (GM) and white matter (WM) changes as compared to healthy controls remains uncertain. Methods: 3D T1-weighted and diffusion tensor images were acquired at a single site in 170 TLE patients (aged 23-74 years) with MRI signs of unilateral hippocampal sclerosis (HS, 77 right) and 111 healthy controls (aged 26-80 years). Global brain (GM, WM, total brain, and cerebrospinal fluid) and regional volumes (ipsi-and contralateral hippocampi), and fractional anisotropy (FA) of 10 tracts (three portions of corpus callosum, inferior longitudinal, inferior frontooccipital and uncinate fasciculi, body of fornix, dorsal and parahippocampalcingulum, and corticospinal tract) were compared between groups as a function of age. Results: There were significant reductions of global brain and hippocampi volumes (greatest ipsilateral to HS), and FA of all 10 tracts in TLE versus controls. For TLE patients, regression lines run in parallel to those from controls for brain volumes and FA (for all tracts except the parahippocampalcingulum and corticospinal tract) versus age across the adult lifespan. Interpretation: These results imply a developmental hindrance occurring earlier in life (likely in childhood/neurodevelopmental stages) rather than accelerated atrophy/ degeneration of most brain structures herein analyzed in patients with TLE.