Age and sex can influence brain iron levels. We studied the influence of these variables on deep gray matter magnetic susceptibilities. In 183 healthy volunteers (44.7 ± 14.2 years, range 20-69, ♀ 49%), in vivo Quantitative Susceptibility Mapping (QSM) at 1.5T was performed to estimate brain iron accumulation in the following regions of interest (ROIs): caudate nucleus (Cd), putamen (Pt), globus pallidus (Gp), thalamus (Th), pulvinar (Pul), red nucleus (Rn), substantia nigra (Sn) and the cerebellar dentate nuclei (Dn). We gauged the influence of age and sex on magnetic susceptibility by specifying a series of Structural Equation Models. The distributions of susceptibility varied in degree across the structures, conforming to histologic findings (Hallgren & Sourander, 1958), with the highest degree of susceptibility in the Gp and the lowest in the Th. Iron increase correlated across several ROIs, which may reflect an underlying age-related process. Advanced age was associated with a particularly strong linear rise of susceptibility in the striatum. Nonlinear age trends were found in the Rn, where they were the most pronounced, followed by the Pul and Sn, while minimal nonlinear trends were observed for the Pt, Th, and Dn. Moreover, sex related variations were observed, so that women showed lower levels of susceptibility in the Sn after accounting for age. Regional susceptibility of the Pul increased linearly with age in men but exhibited a nonlinear association with age in women with a leveling off starting from midlife. Women expected to be post menopause (+51 years) showed lower total magnetic susceptibility in the subcortical gray matter. The current report is consistent with previous reports of age related variations of brain iron, but also adds to the current knowledge by reporting age-related changes in less studied, smaller subcortical nuclei. This is the first in-vivo report to show lower total subcortical brain iron levels selectively in women from midlife, compared to men and younger women. These results encourage further assessment of sex differences in brain iron. We anticipate that age and sex are important co-factors to take into account when establishing a baseline level for differentiating pathologic neurodegeneration from healthy aging. The variations in regional susceptibility reported herein should be evaluated further using a longitudinal study design to determine within-person age related changes.