Dalbergia odorifera T. Chen (Fabaceae) is a woody tree species indigenous to Hainan Island in China. Due to its high medicinal and commercial value, this tree species has been planted over 3500 ha2 in southern China. There is an urgent need for improvement of the D. odorifera germplasm, however, limited information on germplasm collection, conservation, and assessment of genetic resources is available. Therefore, we have built a database of 251 individuals collected across the whole of southern China, which included 42 wild trees and 210 cultivated trees, with the following objectives. (1) Evaluate genetic diversity and population structure of the database using 19 microsatellite markers and (2) develop a core collection for improvement and breeding programs. Totally, the 19 microsatellite markers harbored 77 alleles across the database with the polymorphic information content (PIC) ranging from 0.03 to 0.66. Medium genetic diversity level was inferred by Nei’s gene diversity (0.38), Shannon’s information index (0.65), and observed (0.33) and expected heterozygosity (0.38). Structure analysis showed that four was the optimum cluster size using the model-based Bayesian procedure, and the 251 D. odorifera individuals were grouped into five populations including four pure ones (RP1-4) and one mixed one (MIX) based on their maximum membership coefficients. Among these populations, the expected heterozygosity varied from 0.30 (RP3) to 0.38 (RP4). Analysis of molecular variance (AMOVA) showed 11% genetic variation existed among populations, and moderate population differentiation was inferred by the matrix of pairwise Fst (genetic differentiation among populations), which was in the range of 0.031 to 0.095. Moreover, a core collection of 31 D. odorifera individuals including six wild and 25 cultivated trees was developed, which was only 12.4% of the database but conserved the whole genetic diversity. The results of this study provided additional insight into the genetic structure of the large D. odorifera germplasm, and the core collection will be useful for the efficient and sustainable utilization of genetic resources, as well as efficient improvement in breeding programs.