Seeds directly determine the survival and population size of woody plants, but the genetic basis of seed weight in woody plants remain poorly explored. To identify genetic variations and candidate genes responsible for seed weight in natural woody populations, we investigated the hundred-seed weight of 198 paper mulberry individuals from different areas. Our results showed that the hundred-seed weight of paper mulberry was significantly associated with the bioclimatic variables of sampling sites, which increased from south to north along the latitudinal-temperature gradient. Using 2,414,978 high-quality SNPs from re-sequencing data, the genome-wide association analysis of the hundred-seed weight was performed under three models, which identified 148, 19 and 12 associated genes, respectively. Among them, 25 candidate genes were directly hit by the significant SNPs, including the WRKY transcription factor, fatty acid desaturase, F-box protein, etc. Most importantly, we identified three crucial genetic variations in the coding regions of candidate genes (Bp02g2123, Bp01g3291 and Bp10g1642), and significant differences in the hundred-seed weight were detected among the individuals carrying different genotypes. Further analysis revealed that Bp02g2123 encoding a fatty acid desaturase (FAD) might be a key factor affecting the seed weight and local climate adaptation of woody plants. Furthermore, the genome-wide investigation and expression analysis of FAD genes were performed, and the results suggested that BpFADs widely expressed in various tissues and responded to multiple phytohormone and stress treatments. Overall, our study identifies valuable genetic variations and candidate genes, and provides a better understanding of the genetic basis of seed weight in woody plants.