During aging, intracranial volume remains unchanged and represents maximally attained brain size, while various interacting biological phenomena lead to brain volume loss. Consequently, intracranial volume and brain volume in late life reflect different genetic influences. Our genome-wide association study in 8,175 community-dwelling elderly did not reveal any genome-wide significant associations (p<5*10−8) for brain volume. In contrast, intracranial volume was significantly associated with two loci: rs4273712 (p=3.4*10−11), a known height locus on chromosome 6q22, and rs9915547, tagging the inversion on chromosome 17q21 (p=1.5*10−12). We replicated the associations of these loci with intracranial volume in a separate sample of 1,752 older persons (p=1.1*10−3 for 6q22 and p=1.2*10−3 for 17q21). Furthermore, we also found suggestive associations of the 17q21 locus with head circumference in 10,768 children (mean age 14.5 months). Our data identify two loci associated with head size, with the inversion on 17q21 also likely involved in attaining maximal brain size.