Flame image feature extraction is the basis for boiler combustion monitoring and control. The flame video images of recent research are mainly derived from experimental burners in the laboratory, and few pay attention to the flame images in industrial boilers. The actual industrial boiler flame images differ significantly from the laboratory flame images. Additionally, certain flame image features cannot be captured in the laboratory owing to the limitations of the camera installations. Therefore, a flame image texture feature extraction algorithm based on an industrial boiler is proposed in this paper. The texture features were enhanced using a Gabor filter for the RGB channels of the flame images, and then, the statistics of the texture features were scalarized by a gray-level co-occurrence matrix (GLCM). The data were filtered and downscaled by a data compressor consisting of Gaussian-weighted mean and principal component analysis (PCA) to obtain eight key variables. The extracted eight variables were verified to be effective in characterizing the O2 and NO x contents of flue gas using the mutual information method. The combustion process regression model was constructed using a gated recurrent unit (GRU) on the 8 h combustion data of the boiler, and the predicted mean absolute percentage error (MAPE) for O2 and NO x content in the test set reached 7.5 and 10.2%, respectively. Compared to the conventional methods of direct PCA on images and GLCM plus PCA on images, the MAPE for O2 content prediction was reduced by 12.3 and 7.3%, and the MAPE for NO x content prediction was reduced by 10.5 and 6.1%, respectively. The advantage of the new flame feature based on Gabor-GLCM is suitable for the subsequent analysis and control of an industrial combustion system.