This study numerically investigated the thermoacoustic combustion instability characteristics of a scaled rocket combustor based on a hybrid of the Reynolds-averaged Navier–Stokes and large–eddy simulation method. The turbulence–combustion interactions were treated using flamelet generated manifold approach. An unstable case was simulated with detailed reaction mechanisms (GRI-Mech 3.0). The obtained results agree well with experiment data from Purdue University, in terms of pressure oscillations frequency and power spectral density spectrum. The combustion instability mode was identified to be coupled with the first longitudinal acoustic mode of the combustion chamber by dynamic model decomposition method. According to Rayleigh index analysis, the unstable driving source was found to be located near the combustor step, which was further confirmed by time-averaged flow fields. Detailed three-dimensional vortex ring shedding evolutions at the combustor step were tracked with fine time resolution. Results indicate that the combustion instability arises from periodic vortex ring shedding at the combustor step and interacting with the chamber wall. The unburnt reactants were rolled up by the shedding vortex ring, which would not break up until impact with the chamber wall. Therefore, the mixing performance was significantly enhanced, leading to sudden heat release. Consequently, the thermal energy is added to the acoustic field, and the first longitudinal mode is thus reinforced, giving rise to large amplitude axial velocity oscillations which prompt the generation of the new vortex ring. The results of the present investigation will support the design and development of high-performance rocket engines.