The A-kinase anchoring protein 12 gene (AKAP12) is a scaffold protein, which can target multiple signal transduction effectors, can promote mitosis and cytokinesis and plays an important role in the regulation of growth and development. In our previous study, P1–7 bp (intron 3) and P2–13 bp (3′UTR) indels within the AKAP12 gene significantly influenced AKAP12 gene expression. Therefore, this study aimed to identify the association between these two genetic variations and growth-related traits in Shaanbei white cashmere goats (SBWC) (n = 1405). Herein, we identified two non-linkage insertions/deletions (indels). Notably, we found that the P1–7 bp indel mutation was related to the height at hip cross (HHC; p < 0.05) and the P2–13 bp indel was associated with body weight, body length, chest depth, chest width, hip width, chest circumference and cannon (bone) circumference in SBWC goats (p < 0.05). Overall, the two indels’ mutations of AKAP12 affected growth traits in goats. Compared to the P1–7 bp indel, the P2–13 bp indel is more suitable for the breeding of goat growth traits.