Highly seed vigor (SV) is essential for rice direct seeding (DS). Understanding the genetic mechanism of SV-related traits could contribute to increasing the efficiency of DS. However, only a few genes responsible for SV have been determined in rice, and the regulatory network of SV remains obscure. In this study, the seed germination rate (GR), seedling shoot length (SL), and shoot fresh weight (FW) related to SV traits were measured, and a genome-wide association study (GWAS) was conducted to detect high-quality loci responsible for SV using a panel of 346 diverse accessions. A total of 51 significant SNPs were identified and arranged into six quantitative trait locus (QTL) regions, including one (qGR1-1), two (qSL1-1, qSL1-2), and three (qFW1-1, qFW4-1, and qFW7-1) QTLs associated with GR, SL, and FW respectively, which were further validated using chromosome segment substitution lines (CSSLs). Integrating gene expression, gene annotation, and haplotype analysis, we found 21 strong candidate genes significantly associated with SV. In addition, the SV-related functions of LOC_Os01g11270 and LOC_Os01g55240 were further verified by corresponding CRISPR/Cas9 gene-edited mutants. Thus, these results provide clues for elucidating the genetic basis of SV control. The candidate genes or QTLs would be helpful for improving DS by molecular marker-assisted selection (MAS) breeding in rice.