In this paper, the nonlinear dynamic characteristics of the oxygen-enriched combustion of waste oil biodiesel in semi-industrial furnaces were tested by the power spectrum, phase space reconstruction, the largest Lyapunov exponents, and the 0-1 test method. To express the influences of the system parameters, experiments were carried out under different oxygen content conditions (21%, 25%, 28%, 31%, and 33%). Higher oxygen enrichment degrees contribute to finer combustion sufficiency, which produces flames with high luminance. Flame luminance and temperature can be represented by different gray scale values of flame images. The chaotic characteristics of gray scale time series under different oxygen enrichment degrees were studied. With increased oxygen content, the chaotic characteristics of flame gradually developed from weak chaos to strong chaos. Furthermore, the flame maintained a stable combustion process in a high-temperature region. The stronger the chaotic characteristics of the flame, the better the combustion effect. It can be seen that the change of initial combustion conditions has a great influence on the whole combustion process. The results of several chaotic test methods were consistent. Using chaotic characteristics to analyze the waste oil biodiesel combustion process can digitize the combustion process, find the best combustion state, optimize, and precisely control it.