Studies examining the link between abnormal fetal growth and cardiac changes in childhood have presented conflicting results. We studied the effect of abnormal fetal growth on cardiac morphology and function during childhood, while controlling for body size, composition and postnatal factors. We report on the follow-up of 90 children (median age 5.81 years, IQR 5.67; 5.95) born appropriate for gestational age (AGA, N = 48), small for gestational age (SGA, N = 23), or large for gestational age (LGA, N = 19); SGA and LGA defined as birth weight Z-score < − 2 and > + 2, respectively. We examined the heart using echocardiography, including Doppler and strain imaging, in relation to anthropometrics, body composition, blood pressure, physical activity, and diet. Although groupwise differences in body size decreased during the first year after birth, LGA remained larger at follow-up, with higher lean body mass and BMI, while SGA were smaller. Slight changes in left ventricular diastolic function were present in SGA and LGA, with SGA showing increased mitral diastolic E- and A-wave peak flow velocities, and increased septal E/E′ ratio, and LGA showing larger left atrial volume adjusted for sex and lean body mass. In univariate analyses, lean body mass at follow-up was the strongest predictor of cardiac morphology. We found no groupwise differences at follow-up for ventricular sphericity, cardiac morphology adjusted for lean body mass and sex, or blood pressure, diet, or physical activity. Cardiac morphology is predicted by lean body mass during childhood, even in the setting of abnormal fetal growth. Our results are consistent with a limited effect of fetal programming on cardiac dimensions during childhood. Minor changes in diastolic function are present in both SGA and LGA children, however, the clinical significance of these changes at this stage is likely small.