Stress evaluation plays a pivotal role in the design of material systems, often accomplished through the finite element method (FEM) for intricate structures. However, the substantial costs and time requirements associated with multi-scale FEM analyses have prompted a growing interest in adopting more efficient, machine-learning-driven strategies. This study investigates the utilization of advanced machine learning techniques for predicting local stress fields in composite materials, presenting it as a superior alternative to traditional FEM approaches. The primary objective of this research is to develop a predictive model for stress field maps in composite components featuring diverse configurations of fibers distributed within the matrix. To achieve this, we employ a Convolutional Neural Network (CNN) with a specialized U-Net architecture, enabling the correlation of spatial fiber organization with the resultant von Mises stress field. The CNN model was extensively trained using four distinct data sets, encompassing uniform fibrous structures, non-uniform fibrous structures, irregularly shaped fibrous structures, and a comprehensive combination of these data sets. The trained U-Net models demonstrate exceptional proficiency in predicting von Mises stress fields, yielding impressive structural similarity index scores (SSIM) of 0.977 and mean squared errors (MSE) of 0.0009 on a dedicated test set. This research harnesses 2D cross-sectional imagery to establish a surrogate model for finite element analysis, offering an accurate and efficient approach for predicting stress fields in composite material design, irrespective of geometric complexity or boundary conditions.