Sophora japonica is an important native tree species in northern China, with high ornamental, medicinal, and ecological value. In order to elucidate the genetic resources of ancient S. japonica, 16 simple sequence repeat (SSR) markers were used to evaluate its genetic diversity and population structure and build a core collection of 416 germplasms from the Shandong, Shanxi, and Hebei provinces. A total of 160 alleles were detected, the mean major allele frequency (MAF)was 0.39, and the mean effective number of alleles (Ne) was 4.08. Shannon’s information index (I), the observed heterozygosity (Ho), the expected heterozygosity (He), and the polymorphism information content (PIC) were 1.58, 0.64, 0.74, and 0.70, respectively, indicating relatively high genetic diversity in ancient S. japonica germplasms. Low genetic differentiation coefficient (Fst = 0.04) and frequent gene flow (Nm = 9.74) were found in the tested S. japonica populations, and an analysis of molecular variance (AMOVA) indicated that the genetic variation mainly came from within individuals (84%). A genetic structure and cluster analysis indicated that 416 ancient S. japonica germplasms could be divided into five subgroups, and there were obvious genetic exchanges among different subgroups. A core collection consisting of 104 (25% of the original collection) germplasms was constructed using the R language package Genetic Subsetter version 0.8 based on the stepwise regression method. The retention rates of the number of alleles (Na), Ne, I, He, and PIC were 87.50%, 106.24%, 103.02%, 102.50%, and 102.74%, respectively. The t-test analysis suggested that there were no significant differences between the core collection and the original collection. The principal coordinate analysis (PCoA) showed that the core collection was uniformly distributed within the initial collection and was able to fully represent the genetic diversity of the original collection. These results provide a scientific basis for the conservation and utilization of ancient S. japonica germplasms.