Lesser degrees of terminal duct lobular unit (TDLU) involution, as reflected by higher numbers of TDLUs and acini/TDLU, are associated with elevated breast cancer risk. In rodent models, the insulin-like growth factor (IGF) system regulates involution of the mammary gland. We examined associations of circulating IGF measures with TDLU involution in normal breast tissues among women without precancerous lesions. Among 715 Caucasian and 283 African American (AA) women who donated normal breast tissue samples to the Komen Tissue Bank between 2009 and 2012 (75% premenopausal), serum concentrations of IGF-I and binding protein (IGFBP)-3 were quantified using enzyme-linked immunosorbent assay. Hematoxilyn and eosin-stained tissue sections were assessed for numbers of TDLUs ("TDLU count"). Zero-inflated Poisson regression models with a robust variance estimator were used to estimate relative risks (RRs) for association of IGF measures (tertiles) with TDLU count by race and menopausal status, adjusting for potential confounders. AA (vs. Caucasian) women had higher age-adjusted mean levels of serum IGF-I (137 vs. 131 ng/mL, p = 0.07) and lower levels of IGFBP-3 (4165 vs. 4684 ng/mL, p < 0.0001). Postmenopausal IGFBP-3 was inversely associated with TDLU count among AA (RR = 0.49, 95% CI = 0.28-0.84, p-trend = 0.04) and Caucasian (RR =0.64, 95% CI = 0.42-0.98, p-trend = 0.04) women. In premenopausal women, higher IGF-I:IGFBP-3 ratios were associated with higher TDLU count in Caucasian (RR =1.33, 95% CI = 1.02-1.75, p-trend = 0.04), but not in AA (RR =0.65, 95% CI = 0.42-1.00, p-trend = 0.05), women. Our data suggest a role of the IGF system, particularly IGFBP-3, in TDLU involution of the normal breast, a breast cancer risk factor, among Caucasian and AA women.