In Inertial Confinement Fusion (ICF), the asymmetry of a hot spot is an important influence factor in implosion performance. Neutron penumbral imaging, which serves as an encoded-aperture imaging technique, is one of the most important diagnostic methods for detecting the shape of a hot spot. The detector image is a uniformly bright range surrounded by a penumbral area, which presents the strength distribution of hot spots. The present diagnostic modality employs an indirect imaging technique, necessitating the reconstruction process to be a pivotal aspect of the imaging protocol. The accuracy of imaging and the applicable range are significantly influenced by the reconstruction algorithm employed. We develop a neural network named Fast Fourier transform Neural Network (FFTNN) to reconstruct two-dimensional neutron emission images from the penumbral area of the detector images. The FFTNN architecture consists of 16 layers that include a FFT layer, convolution layer, fully connected layer, dropout layer, and reshape layer. Due to the limitations in experimental data, we propose a phenomenological method for describing hot spots to generate datasets for training neural networks. The reconstruction performance of the trained FFTNN is better than that of the traditional Wiener filtering and Lucy–Richardson algorithm on the simulated dataset, especially when the noise level is high as indicated by the evaluation metrics, such as mean squared error and structure similar index measure. This proposed neural network provides a new perspective, paving the way for integrating neutron imaging diagnosis into ICF.