As an important source of medicine, food, and high-quality wood, Choerospondias axillaris has been widely planted in South China. However, few studies of its growth traits and rhythm and concerning the selection of superior provenances/families were developed. In this study, 77 families from five provenances were studied and grouped. Tree height, diameter at breast height (DBH), and crown width within and between rows were measured. The relevance between growth traits and environmental factors was also analyzed. The results showed the height, DBH, and crown width within and between rows were divided into slow, mean, and fast growth periods, which was significantly related with sunshine hours in earlier months. Next, monthly rainfall showed an extremely positive correlation with the increase in tree height, DBH, and plant volume. Then, the monthly volume increment had a significantly positive correlation with five meteorological factors, excluding sunshine hours with increasing months, while the monthly growth of tree height and crown width within and between rows was significantly negatively correlated with sunshine hours. Based on these results, the superior families No. 15, 76, and 56, which originated from the Raoping provenance, were selected for their fast growth and available adaptation. These results provided the reliable growth rhythm of a young C. axillaris stand, which established a basis for fertilizing and forest tending. During these processes, meteorological factors, especially humidity and sunshine duration, had important effects on growth, implying that the external climate should be paid more attention to promote fast growth and prevent diseases and insect pests. Moreover, selecting superior families was helpful to further breeding and plantation.