Brain scans acquired across large, age-diverse cohorts have facilitated recent progress in establishing normative brain aging charts. Here, we ask the critical question of whether cross-sectional estimates of age-related brain trajectories resemble those directly measured from longitudinal data. We show that age-related brain changes inferred from cross-sectionally mapped brain charts can substantially underestimate actual changes measured longitudinally. We further find that brain aging trajectories vary markedly between individuals and are difficult to predict with population-level age trends estimated cross-sectionally. Prediction errors relate modestly to neuroimaging confounds and lifestyle factors. Our findings provide explicit evidence for the importance of longitudinal measurements in ascertaining brain development and aging trajectories.