Litter size (LS), an important economic trait in livestock, is so complicate that involves many aspects of reproduction, the underlying mechanism of which particularly in goat has always been scanty. To uncover the genetic basis of LS, the genomic sequence of Jining Gray goat groups (one famous breed for high prolificacy in China) with LS 1, 2, and 3 for firstborn was analyzed, obtaining 563.67 Gb sequence data and a total of 31,864,651 high-quality single nucleotide polymorphisms loci were identified. Particularly, the increased heterozygosity in higher LS groups, and large continuous homozygous segments associated with lower LS group had been uncovered. Through an integrated analysis of three popular methods for detecting selective sweeps (Fst, nucleotide diversity, and Tajima's D statistic), 111 selected regions and 42 genes associated with LS were scanned genome wide. The candidate genes with highest selective signatures included KIT, KCNH7, and KMT2E in LS2 and PAK1, PRKAA1, and SMAD9 in LS3 group, respectively. Meanwhile, functional terms of programmed cell death involved in cell development and regulation of insulin receptor signaling pathway were mostly enriched with 42 candidate genes, which also included reproduction related terms of steroid metabolic process and cellular response to hormone stimulus. In conclusion, our study identified novel candidate genes involving in regulation of LS in goat, which expand our understanding of genetic fundament of reproductive ability, and the novel insights regarding to LS would be potentially applied to improve reproductive performance.