Driven by economic interests, adding adulterations in chili powder is a problem which threatens people’s health. Thanks to its nondestructive, rapid, and portable advantages, electronic nose has more potential to be used for adulteration detection than the traditional methods. An approach for identifying the adulterants in chili powder was proposed in this paper. Firstly, an electronic nose system with 10 gas sensors was designed, and then the response images were drawn based on the response signals of the electronic nose. Afterwards, gas features were extracted from those response images by using a histogram of oriented gradients (HOG) algorithm. Finally, an SVM-based identification model was constructed to achieve the identification of plant adulterants in chili powder. The experimental results showed that the identification accuracy of the adulterant categories (almond shell, red beetroot, and tomato peel) based on the HOG features could reach up to 98.3%, and the identification results for adulterant content were 94.2%, 93.3%, and 95%, respectively. Furthermore, in order to compare the efficiency of the proposed identification approach, the widely used model AlexNet was also investigated and discussed.