Central to the investigation of the biomechanics of traumatic brain injury (TBI) and the assessment of injury risk from head impact are finite element (FE) models of the human brain. However, many existing FE human brain models have been developed with simplified representations of the parenchyma, which may limit their applicability as an injury prediction tool. Recent advances in neuroimaging techniques and brain biomechanics provide new and necessary experimental data that can improve the biofidelity of FE brain models. In this study, the CAB-20MSym template model was developed, calibrated, and extensively verified. To implement material heterogeneity, a magnetic resonance elastography (MRE) template image was leveraged to define the relative stiffness gradient of the brain model. A multi-stage inverse FE (iFE) approach was used to calibrate the material parameters that defined the underlying non-linear deviatoric response by minimizing the error between model-predicted brain displacements and experimental displacement data. This process involved calibrating the infinitesimal shear modulus of the material using low-severity, low-deformation impact cases and the material non-linearity using high-severity, high-deformation cases from a dataset of in situ brain displacements obtained from cadaveric specimens. To minimize the geometric discrepancy between the FE models used in the iFE calibration and the cadaveric specimens from which the experimental data were obtained, subject-specific models of these cadaveric brain specimens were developed and used in the calibration process. Finally, the calibrated material parameters were extensively verified using independent brain displacement data from 33 rotational head impacts, spanning multiple loading directions (sagittal, coronal, axial), magnitudes (20–40 rad/s), durations (30–60 ms), and severity. Overall, the heterogeneous CAB-20MSym template model demonstrated good biofidelity with a mean overall CORA score of 0.63 ± 0.06 when compared to in situ brain displacement data. Strains predicted by the calibrated model under non-injurious rotational impacts in human volunteers (N = 6) also demonstrated similar biofidelity compared to in vivo measurements obtained from tagged magnetic resonance imaging studies. In addition to serving as an anatomically accurate model for further investigations of TBI biomechanics, the MRE-based framework for implementing material heterogeneity could serve as a foundation for incorporating subject-specific material properties in future models.