Accurate combustion state recognition of flame images not only plays an important role in social security, but also contributes to increasing thermal efficiency and product quality. To improve the accuracy of feature extraction and achieve the combustion state recognition, a novel method based on radial Chebyshev moment invariants (RCMIs) and an improved firefly algorithm-wavelet support vector machine (IFA-WSVM) model is proposed. Firstly, the potential flame pixels and the potential flame contour are obtained in the pre-processing phase. Then, the rotation, translation and scaling (RTS) invariants of radial Chebyshev moments are derived. Combing the region and contour moments, the RCMIs of pre-processed and edge images are calculated to construct multi-feature vectors. To enhance the recognition performance, an IFA-WSVM model is built, where the IFA is applied to search the best parameters of WSVM. Then, the IFA-WSVM model is used to recognize the combustion state. Finally, the result for case studies show that the proposed method is superior to methods based on HMIs and ZMIs, achieving the highest rate of 99.07% in real time. The IFA algorithm also outperforms other benchmark algorithms. Even for the images transformed by RTS and small size of training sets, the proposed method continues to exhibit the best performance.