It is imperative to study sex differences in brain morphology and function. However, there are major observable and unobservable confounding factors that can contribute to the estimated differences. Males have larger head sizes than females. Head size differences not only act as a confounding factor in studying sex differences in the brain, but also impact its anatomy and functioning. In this work, we seek to disentangle the effect of head size from sex in studying sex differentiated aging trajectories, its relation to canonical functional networks and cytoarchitectural classes, brain allometry, cognition. Using the UK Biobank (UKBB) neuroimaging data (N = 35,732 participants, 19,281 females, 44-82 years of age), we created a subsample (N = 11,294) where females (N = 5,657) and males were matched by their total intracranial volume (TIV) and age, a subsample that maintains the UKBB sample distribution, one matched only by age, and one that exaggerated the TIV difference between sexes. We then modeled the aging trajectories at both regional and vertex-wise levels in the four subsamples, and compared the estimations of the models. Our results show that when females and males have the same head size, the overall sex estimations tend towards zero, suggesting that most of the variability results from head size differences. Our approach also revealed bidirectional sex differences in brain neuroanatomy previously masked by the effect of head size. Further, the scaling relationship between regional and total brain volume remains fairly consistent across the lifespan and is not sex differentiated overall. We evaluated how the results of cognitive tests with perceived sex differences are influenced and explained by head size and found that although the correlation between TIV and cognitive scores is low, the matching process changes the direction of the effect sizes of differences between sexes in “verbal and numerical reasoning” and “working memory” cognitive domains. Taken together, employing a matching approach that is widely used in causal modeling studies, we provide new evidence for disentanglement of sex differences in the brain from head size as a biological confound.