Fire image monitoring systems are being applied to more and more fields, owing to their large monitoring area. However, the existing image processing-based fire detection technology cannot effectively make real-time fire warning in actual scenes, and the relevant fire recognition algorithms are not robust enough. To solve the problems, this paper tries to extract and classify image features for fire recognition based on convolutional neural network (CNN). Specifically, the authors set up the framework of a fire recognition system based on fire video images (FVIFRS), and extracted both static and dynamic features of flame. To improve the efficiency of image analysis, a Gaussian mixture model was established to extract the features from the fire smoke movement areas. Finally, the CNN was improved to process and classify the fire feature maps of the CNN. The proposed algorithm and model were proved to be feasible and effective through experiments.