The purpose of this study was to investigate the effects of non-genetic factors on the growth and development performance of Inner Mongolia white cashmere goats (Erlanghan type), such as birth weight (BW), weaning weight (WW), 6-month weight (6 WT), 12-month weight (12 WT), body height (BH), and body length (BL), and wool production performance, such as cashmere fineness (CF), cashmere thickness (CT), and cashmere yield (CY). The research objects were 4654 kids produced by 45 buck goats and 2269 doe goats in the Erlang Mountain Ranch of Beiping Textile Co., Ltd., Inner Mongolia, from 2020 to 2023. Based on the generalized linear model, ANOVA was used to analyze the effects of non-genetic factors, such as birth year (Y), birth month (M), sex (S), birth type (T), birth herd (H), assay flock (F), age at measurement (MA), and the age of doe goats at lambing (DLA), on growth and development traits and cashmere traits. The results show that the birth weight (BW), weaning weight (WW), 6-month weight (6 WT), 12-month weight (12 WT), body length (BL), body height (BH), chest depth (CD), chest width (CW), chest circumference (CC), cannon circumference (CNC), wool length (WL), and cashmere yield (CY) of buck goats were significantly higher than those of doe goats (p < 0.01), and the fineness of the cashmere produced by doe goats was significantly finer than that produced by buck goats (p < 0.01). The birth weight, weaning weight, and 6-month weight of single kids were significantly higher than those of multiple kids (p < 0.01), but the effect on the 12-month weight was not significant (p > 0.05). The age of doe goats at lambing had significant effects on birth weight, weaning weight, and 6-month weight (p < 0.01). Assay flock and age at measurement had significant effects on cashmere fineness, cashmere thickness, and cashmere yield (p < 0.01). This study will provide a basis for the scientific breeding and management of cashmere goats and lay a foundation for the setting of fixed effects in the genetic evaluation model of Inner Mongolia white cashmere goats (Erlangshan type).