The hypothalamus and pituitary regulate, amongst other functions, third order endocrine systems, and their volumes have been associated with normal and pathological outcomes. Yet, there are very few studies that examine their combined structural variations in vivo. This is due, in part, to their small size and a lack of comprehensive image segmentation protocols. In the current project we acquired high-resolution T1- (1mm isotropic) and T2-weighted (0.4mm in plane resolution) 3T magnetic resonance images (MRI) of the hypothalamus and pituitary gland, as well as salivary estradiol and testosterone from 31 (17M, 14F) young healthy adults. Women reported oral contraceptive use. Image preprocessing included non-uniformity correction, signal intensity normalization and standard stereotaxic space registration. We applied a comprehensive manual segmentation protocol of the whole hypothalamus, with detailed segmentation of the pituitary stalk, the anterior and posterior pituitary gland, and the posterior bright spot. We also propose a novel medial-lateral hypothalamic parcellation into medial preoptic, periventricular (PVN), and lateral hypothalamic regions. The protocol yielded good inter- (range: 0.78-0.92) and intra-rater (range: 0.79-0.94) Dice kappa overlap coefficients. We detected sex differences of the whole hypothalamus and each hemisphere, and a trend for the right preoptic region to be larger in males than in females, with a moderate effect size. Sex differences were maintained or enhanced when covarying for estradiol, but not when covarying for testosterone. In addition, testosterone was associated with the volume of the PVN, but only in women. In summary, these results suggest that there are morphometric differences at the level of the pituitary and hypothalamus that are likely driven by central regulation of gonadal hormones. The here described protocol allows the structural investigation of neuroendocrine effects in the central nervous system.