Deep neural networks trained on jet images have been successful in classifying different kinds of jets. In this paper, we identify the crucial physics features that could reproduce the classification performance of the convolutional neural network in the top jet vs. QCD jet classification. We design a neural network that considers two types of substructural features: two-point energy correlations, and the IRC unsafe counting variables of a morphological analysis of jet images. The new set of IRC unsafe variables can be described by Minkowski functionals from integral geometry. To integrate these features into a single framework, we reintroduce two-point energy correlations in terms of a graph neural network and provide the other features to the network afterward. The network shows a comparable classification performance to the convolutional neural network. Since both networks are using IRC unsafe features at some level, the results based on simulations are often dependent on the event generator choice. We compare the classification results of Pythia 8 and Herwig 7, and a simple reweighting on the distribution of IRC unsafe features reduces the difference between the results from the two simulations.