Here we present an innovative approach for detecting threat materials within a sealed container by integrating tagged fast neutron activation analysis with Explainable Artificial Intelligence (XAI). Two AI models, a Feed-Forward Neural Network (FFNN) and a Convolutional Neural Network (CNN), were developed to analyze the emitted gamma rays to identify materials like explosives and drugs based on depth profiles of carbon, nitrogen, and oxygen concentrations. XAI was applied to make the models' decision-making process transparent. The method is adaptable to various spectrometric analyses. We demonstrate its effectiveness using data obtained by the Rapidly Relocatable Tagged Neutron Inspection System (RRTNIS), which is irreplaceable for inspecting sealed cargo containers, despite challenges such as variable material placement, background noise, and shielding effects. Our approach successfully locates and categorizes threat materials, both alone and within surrounding materials, at various locations within sealed cargo containers.