The traditional analysis method of engine combustion cycle variation is a statistical method based on a small amount of data. In essence, the obtained cycle variation is random data. In order to reveal the dynamic nature of the cyclical changes during the combustion of a shale gas engine, a nonlinear dynamics method was used to study the stability of the combustion process. The motion law of the phase space trajectory is analyzed, the influence of the shale gas composition on the trajectory distribution is analyzed, the return mapping point of the average indicated pressure in the cylinder is discussed. The relationship between adjacent combustion characteristic parameters is studied; the chaotic characteristics of the shale gas engine combustion process are discussed. The results show that during the working process of the shale gas engine, the in-cylinder pressure shows a similar quasi-periodic state in the entire phase space, and the working process has a certain chaotic law; with the increase of the CH4, N2 and CO2 content in the shale gas, the combustion cycle variation increases, and the randomness of the engine working process increases. The phase space trajectory shows a monotonously increasing distribution of Poincaré mapping points on the ∑XY+ section. With the increase of the combustion cycle, the linear relationship of the scattered points gradually increases, and the randomness of the combustion process increases. The return map points of the engine combustion characteristic parameters are distributed in a cluster. When the CH4 content increases, the distribution range of the average indicated pressure return map points increases. With the increase of N2 and CO2 content, abnormal combustion phenomena such as partial combustion or misfire occur during the engine combustion process, the uncertainty of the combustion process increases, and the combustion stability decreases. With the increase of engine speed, the correlation dimension and the maximum Lyapunov exponent increase, the randomness of the combustion process increases, and the chaotic characteristics of the engine working process are obvious; the time series of the cylinder pressure is more sensitive to the content of inert gas. With the increase of N2 and CO2 content in the gas, the correlation dimension and the maximum Lyapunov exponent increase significantly, the complexity of the phase space trajectory increases, and the chaotic characteristics become more obvious.