The massive analysis of Rutherford Backscattering Spectrometry (RBS) data is complex. When the data is processed manually, it requires a long time of an experienced person. Artificial Neural Networks (ANN) can analyze, speed up, and automate data processing. In fact, after training, the ANN processes one RBS spectrum in a fraction of a second with the advantage of keeping the consistency over the whole set of spectra. Our group used ANN to process a large set of RBS spectra from the inner walls of the vacuum chamber of the W7-X fusion reactor. In this work, we used a perturbation-based method to study the local explanations of the neural network predictions. In this method, we apply small perturbations to the inputs. Then, the outputs’ variations are evaluated. Thereby activation maps were created to visualize how sensitive the ANN is to perturbations. The activation maps enable the identification of the parts of the spectrum the neural network is getting information to make predictions. Therefore, we can better understand the behavior of the machine learning model and verify if the neural network learned the features of the spectra similarly to humans.