Simple Summary: Genome-wide association studies found that the PR/SET Domain 6 (PRDM6) gene mutation was associated with bone development, bone density, and body mass index. This study found a 12 bp deletion variation within the PRDM6 gene in Shaanbei white cashmere goats in a large sample size (n = 1044). This variation polymorphism was associated with multiple goat growth traits in the yearling period, including heart girth (p = 0.027), cannon circumference (p = 0.008), chest depth (p = 2.10 × 10 −5 ), chest width (p = 0.004), body height (p = 0.032), body length (p = 0.044), and hip-width (p = 0.014). The effects of the 12 bp variation were found to make no difference on adult goat growth traits. Taken together, these results demonstrate that the 12 bp deletion variant plays an important role in the early growth and development of goats and could be considered as a useful and effective molecular marker for goat breeding selection in the growth stage.Abstract: By genome-wide association studies, the PRDM6 gene has been shown to affect multiple, apparently unrelated inherited traits, including bone density and body mass index. Therefore, it is considered a potentially pleiotropic gene. In this study, we identified a 12 bp deletion variant (NC_030814.1:rs651603667, g: 79985625-79985636delTTGACTGATCCA) within the PRDM6 gene in a large sample (SBWC goats; n = 1044). All goat samples were collected in Shaanxi province in July 2018. The frequency of the wt allele was higher than the frequency of the del allele, and this mutation polymorphism confirmed to be consistent with the Hardy-Weinberg equilibrium (p > 0.05). Further results showed that in a group of goats in the yearling period (18 months old, n = 567), this deletion variant of the PRDM6 gene was associated with heart girth (p = 0.027), cannon circumference (p = 0.008), chest depth (p = 2.10 × 10 −5 ), chest width (p = 0.004), body height (p = 0.032), body length (p = 0.044) and hip-width (p = 0.014). For adult SBWC goats (36 months old, n = 477), the effects of the 12 bp variation on growth-related traits were found to make no difference. These findings show that the 12 bp deletion within the goat PRDM6 gene plays an important role in the early growth and development of goats. Using the 12 bp mutation, breeders can quickly and effectively select excellent individual goats at an early stage.