Recent animal work has documented a two-way interconnection between calvarial bone marrow and the brain. Bone marrow adiposity may therefore have direct implications for brain health, yet the lack of in vivo studies in humans limits our understanding of such interplay. Here, we exploit large magnetic resonance imaging (MRI) datasets to study bone marrow adiposity and its association with brain and body traits. We trained an artificial neural network to localise calvarial bone marrow in T1-weighted 3T MRI head scans. Validation in two independent samples confirmed high accuracy of our approach to semi-quantitatively measure calvarial bone marrow adiposity in vivo. Our analyses revealed sex-dimorphic age trajectories in line with earlier work from non-calvarial bone marrow. Next, we studied the genetic architecture of the calvarial bone marrow and revealed its high heritability in single-nucleotide polymorphism and twin data. We identified 41 genetic loci significantly associated with the calvarial bone marrow, including a sex-specific genetic effect of the estrogen receptor locus. By integrating mapped genes with existing bone marrow single-cell RNA sequencing data, we revealed patterns of adipogenic lineage differentiation and lipid loading. Finally, we identified significant genetic correlation with several heritable human traits, including general cognitive ability and Parkinson′s disease. Our method for quantifying calvarial bone marrow adiposity from MRI head scans enables a significant advance in our understanding of the genetic and cellular drivers of bone marrow adiposity in humans. This contributes to the groundwork needed to elucidate its function and creates an opportunity to study its effects on other organs such as bone, blood, and brain.