This study investigates the application of machine learning models to predict time-evolving stress fields in complex three-dimensional structures trained with full-scale finite element simulation data. Two novel architectures, the Multi-Decoder CNN (MUDE-CNN) and the Multiple Encoder-Decoder Model with Transfer Learning (MTED-TL), were introduced to address the challenge of predicting the progressive and spatial evolutional of stress distributions around defects. The MUDE-CNN leveraged a shared encoder for simultaneous feature extraction and employed multiple decoders for distinct time frame predictions, while MTED-TL progressively transferred knowledge from one encoder-decoder block to another, thereby enhancing prediction accuracy through transfer learning. These models were evaluated to assess their accuracy, with a particular focus on predicting temporal stress fields around an additive manufacturing-induced isolated pore, as understanding such defects is crucial for assessing mechanical properties and structural integrity in materials and components fabricated via additive manufacturing. The temporal model evaluation demonstrated MTED-TL's consistent superiority over MUDE-CNN, owing to transfer learning's advantageous initialization of weights and smooth loss curves. Furthermore, an autoregressive training framework was introduced to improve temporal predictions, consistently outperforming both MUDE-CNN and MTED-TL. By accurately predicting temporal stress fields around AM-induced defects, these models can enable real-time monitoring and proactive defect mitigation during the fabrication process. This capability ensures enhanced component quality and enhances the overall reliability of additively manufactured parts.