In recent years, the number of machine learning applications (especially those involving deep learning) applied to predicting and discovering material properties has been increasing. This paper is based on using microstructure and carbon content to train machine learning models to predict the residual stress of carburized steel. First, a semantic segmentation model of the material organization structure (SegModel-MOS) was constructed based on the AlexNet network and initially trained on the PASCAL VOC2012 dataset. Then, the trained model was fine-tuned on an enhanced homemade dataset consisting of optical microstructures. The experimental results show that SegModel-MOS can distinguish acicular martensite, retained austenite, and lath martensite in microstructures. Finally, we used both support vector machine (SVM) and decision tree (DT) algorithms to establish a mapping relationship between the microstructure, carbon content, and residual stress to predict the residual stress of steel from its microstructure and carbon content. The experiments verified that the prediction model constructed in this study exhibits high accuracy and can directly predict residual stress without requiring any long-term measurements. Thus, the developed model provides a new approach to the study of residual stress in steel.