Modern cosmological research in large-scale structure has witnessed an increasing number of machine-learning applications. Among them, convolutional neural networks (CNNs) have received substantial attention due to their outstanding performance in image classification, cosmological parameter inference, and various other tasks. However, many models based on CNNs are criticized as “black boxes” due to the difficulties in relating their outputs intuitively and quantitatively to the cosmological fields under investigation. To overcome this challenge, we present the Cosmological Correlator Convolutional Neural Network (C3NN)—a fusion of CNN architecture and cosmological N-point correlation functions (NPCFs). We demonstrate that its output can be expressed explicitly in terms of the analytically tractable NPCFs. Together with other auxiliary algorithms, we can open the “black box” by quantitatively ranking different orders of the interpretable outputs based on their contribution to classification tasks. As a proof of concept, we demonstrate this by applying our framework to a series of binary classification tasks using Gaussian and log-normal random fields and relating its outputs to the NPCFs describing the two fields. Furthermore, we exhibit the model’s ability to distinguish different dark energy scenarios (w
0 = −0.95 and −1.05) using N-body simulated weak-lensing convergence maps and discuss the physical implications coming from their interpretability. With these tests, we show that C3NN combines advanced aspects of machine learning architectures with the framework of cosmological NPCFs, thereby making it an exciting tool to extract physical insights in a robust and explainable way from observational data.