Achieving ignition and stable combustion in scramjet combustors, which have residence times of milliseconds, requires effective flame stabilization. Previous studies have explored stable forced ignition and flame holding using various igniters, such as micro-rocket and plasma jet torches. This study investigated the forced ignition and combustion instability of a scramjet combustor with a micro-rocket torch. Although the micro-rocket torch successfully ignited the combustor in previous experiments, combustion instabilities were observed in the cavity flame holder through OH* chemiluminescence images. To understand the mechanisms behind forced ignition and combustion instability, dynamic mode decomposition (DMD) was employed. DMD is effective for analyzing combustion dynamics because it extracts spatiotemporally coherent structures from high-dimensional data. The OH* chemiluminescence images from prior experiments were analyzed using DMD to identify the dominant modes associated with the combustion instability. Power spectrum density analysis revealed that low-frequency instabilities (100–400 Hz) were caused by fuel injector flame feedback, very low frequencies (15–50 Hz) were due to torch gas injector flame feedback, and high frequencies (over 1000 Hz) resulted from interference between the cavity flame holder and the flow field. These findings provide insights into the complex flow dynamics within the cavity flame holder and shear layer, thereby enhancing the understanding of combustion instability mechanisms in scramjet engines equipped with micro-rocket torches.