Altered brain development is a common feature of the neurological sequelae of complex congenital heart disease (CHD). These alterations include abnormalities in brain size and growth that begin prenatally and persist postnatally. However, the longitudinal trajectory of changes in brain volume from the prenatal to postnatal environment have not been investigated. We aimed to evaluate the trajectory of brain growth in a cohort of patients with complex CHD (n = 16) and healthy controls (n = 15) to test the hypothesis that patients with complex CHD would have smaller total brain volume (TBV) prenatally, which would become increasingly prominent by three months of age. Participants underwent fetal magnetic resonance imaging (MRI) at a mean of 32 weeks gestation, a preoperative/neonatal MRI shortly after birth, a postoperative MRI (CHD only), and a 3-month MRI to evaluate the trajectory of brain growth. Three-dimensional volumetric analysis was applied to the MRI data to measure TBV, as well as tissue-specific volumes of the cortical gray matter (CGM), white matter (WM), subcortical (deep nuclear) gray matter (SCGM), cerebellum, and cerebrospinal fluid (CSF). A random coefficients model was used to investigate longitudinal changes in TBV and demonstrated an altered trajectory of brain growth in the CHD population. The estimated slope for TBV from fetal to 3-month MRI was 11.5 cm3 per week for CHD infants compared to 16.7 cm3 per week for controls (p = 0.0002). Brain growth followed a similar trajectory for the CGM (p < 0.0001), SCGM (p = 0.002), and cerebellum (p = 0.005). There was no difference in growth of the WM (p = 0.30) or CSF (p = 0.085). Brain injury was associated with reduced TBV at 3-month MRI (p = 0.02). After removing infants with brain injury from the model, an altered trajectory of brain growth persisted in CHD infants (p = 0.006). These findings extend the existing literature by demonstrating longitudinal impairments in brain development in the CHD population and emphasize the global nature of disrupted brain growth from the prenatal environment through early infancy.