The inbred “SJ11-3” pepper was cultured in yellow brown soil, paddy soil, fluvo-aquic soil, and pastoral soil, and the factors affecting the absorption of trace elements and fruit quality were analyzed. The results showed that the physicochemical properties of the soils were significantly different, which led to differences in the nutritional quality of pepper fruits. The pH value had a significant effect on the absorption of trace elements in pepper. The increase of pH promoted the absorption of magnesium and molybdenum but inhibited the absorption of zinc, copper, manganese, and iron. The stepwise multivariable regression analysis showed that the amount of molybdenum in soil was the main factor affecting the total amino acid content of pepper. Total nitrogen, zinc, and copper were the main factors that contributed to the soluble sugar content of pepper, and the available potassium was the major determinant of the vitamin C content of pepper. This study provides new insight on the pepper fruit quality grown on different types of soil with varying levels of trace elements.