Seed vigor is a complex trait, including the seed germination, seedling emergence, and growth, as well as seed storability and stress tolerance, which is important for direct seeding in rice. Seed vigor is established during seed development, and its level is decreased during seed storage. Seed vigor is influenced by genetic and environmental factors during seed development, storage, and germination stages. A lot of factors, such as nutrient reserves, seed dying, seed dormancy, seed deterioration, stress conditions, and seed treatments, will influence seed vigor during seed development to germination stages. This review highlights the current advances on the identification of quantitative trait loci (QTLs) and regulatory genes involved in seed vigor at seed development, storage, and germination stages in rice. These identified QTLs and regulatory genes will contribute to the improvement of seed vigor by breeding, biotechnological, and treatment approaches.