The Booroola fecundity (FecB) gene is a major fertility-related gene first identified in Booroola sheep. Numerous studies have investigated whether the FecB gene is a major fecundity gene in goats or whether there are other genes that play a critical role in goat fertility. Nevertheless, little attention has been paid to the role of the FecB gene in the body morphometric traits of goats, despite the positive relationship discerned between litter size and growth. We identified five copy number variations (CNVs) within the FecB gene in 641 goats, including 318 Shaanbei white cashmere (SBWC) goats, 203 Guizhou Heima (GZHM) goats, and 120 Nubian goats, which exhibited different distributions among these populations. Our results revealed that these five CNVs were significantly associated with goat morphometric traits (p < 0.05). The normal type of CNV3 was the dominant type and displayed superior phenotypes in both litter size and morphometric traits, making it an effective marker for goat breeding. Consequently, LD blocks in the region of 10 Mb upstream and downstream from FecB and potential transcription factors (TFs) that could bind with the CNVs were analyzed via bioinformatics. Although no significant LD block was detected, our results illustrated that these CNVs could bind to growth-related TFs and indirectly affect the growth development of the goats. We identified potential markers to promote litter size and growth, and we offer a theoretical foundation for further breeding work.