Previous research has demonstrated reduction in cortical and subcortical, including basal ganglia (BG), gray matter volumes (GMV) in individuals with attention deficit hyperactivity disorder (ADHD), a neurodevelopmental condition that is more prevalent in males than in females. However, the volumetric deficits vary across studies. Whether volumetric reductions are more significant in males than females; to what extent these neural markers are heritable and relate to cognitive dysfunction in ADHD remain unclear. To address these questions, we followed published routines and performed voxel-based morphometry analysis of a data set (n = 11,502; 5,464 girls, 9–10 years) curated from the Adolescent Brain Cognition Development project, a population-based study of typically developing children. Of the sample, 634 and 2,826 were identified as monozygotic twins and dizygotic twins/siblings, respectively. In linear regressions, a cluster in the hypothalamus showed larger GMV, and bilateral caudate and putamen, lateral orbitofrontal and occipital cortex showed smaller GMVs, in correlation with higher ADHD scores in girls and boys combined. When examined separately, boys relative to girls showed more widespread (including BG) and stronger associations between GMV deficits and ADHD scores. ADHD traits and the volumetric correlates demonstrated heritability estimates (a2) between 0.59 and 0.79, replicating prior findings of the genetic basis of ADHD. Further, ADHD traits and the volumetric correlates (except for the hypothalamus) were each negatively and positively correlated with N-back performance. Together, these findings confirm volumetric deficits in children with more prominent ADHD traits. Highly heritable in both girls and boys and potentially more significant in boys than in girls, the structural deficits underlie diminished capacity in working memory and potentially other cognitive deficits in ADHD.