Objective: The aim of our study is to explore whether the cerebral growth is delayed in fetuses with congenital heart defects (CHD) in the second and early third trimester. Methods: A prospective cohort study was conducted in 77 CHD cases, with 75 healthy controls. 3D cerebral volume acquisition was performed sequentially. The volumes of the fetal hemicerebrum and extracerebral fluid were compared by linear regression analysis, and the Sylvian fissure was measured. Results: Between 19 and 32 weeks of gestation, 158 measurements in cases and 183 measurements in controls were performed (mean 2.2/subject). The volume growth of the hemicerebrum (R2 = 0.95 vs. 0.95; p = 0.9) and the extracerebral fluid (R2 = 0.84 vs. 0.82, p = 0.9) were similar. Fetuses with abnormal oxygen delivery to the brain have a slightly smaller brain at 20 weeks of gestation (p = 0.02), but this difference disappeared with advancing gestation. CHD cases demonstrated a slightly shallower Sylvian fissure (mean ratio 0.146 vs. 0.153; p = 0.004). Conclusions: Our study shows no differences in cerebral growth, studied in an unselected cohort, with successive cases of isolated CHD. Even in the severest CHD cases, cerebral size is similar in the early third trimester. The cause and meaning of a shallower Sylvian fissure is unclear; possibly, it is a marker for delayed cerebral maturation or it might be an expression of decreasing amount of extracerebral fluid.