In natural aquatic environments, the existence of colored dissolved organic matter (CDOM), suspended particles, and colloids can cause scattering and reflection of light and even emit fluorescence itself. Such interference negatively impacts algal fluorescence, further making it unreliable to measure the algal concentration using three-dimensional excitation–emission matrix (3D-EEM) fluorescence spectroscopy. In this study, we proposed a novel algal fluorescence anti-interference network (AFAI-Net) based on a convolutional neural network. The main procedure of this model can be divided into two parts: (1) to quickly determine if there is an interference of CDOM or turbidity in the detected algal samples; (2) to correct the interfered samples and output the fluorescent components of the algae. We trained the model using the 3D-EEMs of pure algal samples (non-interfered) and mixed samples of algae and CDOM or turbidity (interfered); as a result, the well-trained model achieved a total classification accuracy of 96.82%, and the RMSE of CDOM and turbidity removal fitting effects were 0.2274 and 0.3423, respectively. Compared with the non-negative weighted least squares (NNLS) regression analysis method, using the CNN model for CDOM correction resulted in 13.11%, 0.65%, and 5.69% reductions in the average deviation rate for PD, PG, and CM, respectively. Furthermore, the spectra corrected by the model predicted algal densities that were closer to the true algal densities. This study provides a new way to remove non-algal factors that affect algal fluorescence spectra in water bodies, which is beneficial to monitoring eutrophication and red tide in aquatic systems.