Timing of germination determines whether a new plant life cycle can be initiated; therefore, appropriate dormancy and rapid germination under diverse environmental conditions are the most important features for a seed. However, the genetic architecture of seed dormancy and germination behavior remains largely elusive. In the present study, a linkage analysis for seed dormancy and germination behavior was conducted using a set of 146 chromosome segment substitution lines (CSSLs), of which each carries a single or a few chromosomal segments of Nipponbare (NIP) in the background of Zhenshan 97 (ZS97). A total of 36 quantitative trait loci (QTLs) for six germination parameters were identified. Among them, qDOM3.1 was validated as a major QTL for seed dormancy in a segregation population derived from the qDOM3.1 near-isogenic line, and further delimited into a genomic region of 90 kb on chromosome 3. Based on genetic analysis and gene expression profiles, the candidate genes were restricted to eight genes, of which four were responsive to the addition of abscisic acid (ABA). Among them, LOC_Os03g01540 was involved in the ABA signaling pathway to regulate seed dormancy. The results will facilitate cloning the major QTLs and understanding the genetic architecture for seed dormancy and germination in rice and other crops.