Japanese chestnut (Castanea crenata Sieb. et Zucc.), the only fruit tree species domesticated in Japan, has been cultivated alongside natural stands since prehistorical times. Understanding the genetic diversity of this species and the relationships between cultivated and wild chestnut is important for clarifying its breeding history and determining conservation strategies. We assessed 3 chestnut cultivar populations and 29 wild chestnut populations (618 accessions). Genetic distance analysis revealed that wild populations in the Kyushu region are genetically distant from other populations, whereas other wild and cultivar populations are comparatively similar. Assignment tests suggested that cultivars were relatively similar to populations from central to western Honshu. Bayesian structure analyses showed that wild individuals were roughly classified according to geographical distribution along the Japanese archipelago, except that some wild individuals carried the genetic cluster prevalent in cultivars. Parentage analyses between cultivars and wild individuals identified 26 wild individuals presumed to have a parent–offspring relationship with a cultivar. These results suggested that the genetic structure of some wild individuals in natural stands was influenced by gene flow from cultivars. To conserve wild individuals carrying true “wild” genetic clusters, these individuals should be collected and preserved by ex situ conservation programs.