The impact of variation in mating system on genetic diversity is a well-debated topic in evolutionary biology. The diploid sexual race of Cyrtomium falcatum (Japanese holly fern) shows mating system variation, i.e., it displays two different types of sexual expression (gametangia formation) in gametophytes: mixed (M) type and separate (S) type. We examined whether there is variation in the selfing rate among populations of this species, and evaluated the relationship between mating system, genetic diversity and effective population size using microsatellites. In this study, we developed eight new microsatellite markers and evaluated genetic diversity and structure of seven populations (four M-type and three S-type). Past effective population sizes (Ne) were inferred using Approximate Bayesian computation (ABC). The values of fixation index (FIS), allelic richness (AR) and gene diversity (h) differed significantly between the M-type (FIS: 0.626, AR: 1.999, h: 0.152) and the S-type (FIS: 0.208, AR: 2.718, h: 0.367) populations (when admixed individuals were removed from two populations). Although evidence of past bottleneck events was detected in all populations by ABC, the current Ne of the M-type populations was about a third of that of the S-type populations. These results suggest that the M-type populations have experienced more frequent bottlenecks, which could be related to their higher colonization ability via gametophytic selfing. Although high population differentiation among populations was detected (FST = 0.581, F’ST = 0.739), there was no clear genetic differentiation between the M- and S-types. Instead, significant isolation by distance was detected among all populations. These results suggest that mating system variation in this species is generated by the selection for single spore colonization during local extinction and recolonization events and there is no genetic structure due to mating system.