Neolamarckia cadamba (Roxb.) Bosser is a tropical evergreen broadleaf tree species that could play an important role in meeting the increasing demand for wood products. However, multi-level genetic variation and selection efficiency for growth traits in N. cadamba is poorly characterized. We therefore investigated the efficiency of early selection in N. cadamba by monitoring the height (HT), diameter at breast height (DBH), and tree volume (V) in 39 half-sib families from 11 provenances at ages 2, 3, 4, 5, and 6 years in a progeny test. Age-related trends in growth rate, genetic parameters in multi-level, efficiency of early selection, and realized gain in multi-level for growth traits were analyzed. The result showed that genetic variation among families within provenances was higher than that among provenances. The estimated individual heritability values for the growth traits ranged from 0.05 to 0.26, indicating that the variation of growth traits in N. cadamba was subject to weak or intermediate genetic control. The age–age genetic correlations for growth traits were always positive and high (0.51–0.99), and the relationships between the genetic/phenotypic correlations and the logarithm of the age ratio (LAR) were described well by linear models (R2 > 0.85, except the fitting coefficient of genetic correlation and LAR for HT was 0.35). On the basis of an early selection efficiency analysis, we found that it is the best time to perform early selection for N. cadamba at age 5 before half-rotation, and the selection efficiencies were 157.28%, 151.56%, and 127.08% for V, DBH, and HT, respectively. Higher realized gain can be obtained by selecting superior trees from superior families. These results can be expected to provide theoretical guidance and materials for breeding programs in N. cadamba and can even be a reference for breeding strategies of other fast-growing tree species.