Non-premixed swirl combustion has been widely used in pieces of industrial combustion equipment such as industrial boilers, furnaces, and certain specific gas turbine combustors. In recent years, the combustion instability of non-premixed swirl flames has begun receiving attention, yet there is still a lack of related research in academia. Therefore, in this study, we conducted experimental research on a swirl stabilized gas flame model combustor and studied the heat release response characteristics of the swirl combustor through the flame transfer function. Firstly, the flame transfer function (FTF) was measured under different inlet velocities and equivalence ratios, and the experimental results showed that the FTF gain curve of the non-premixed swirl flame exhibited a significant “bimodal” shape, with the gain peaks located around 230 Hz and 330 Hz, respectively. Secondly, two oscillation modes of the flame near the two gain peaks were identified (the acoustic induced vortex mode Mv and the thermoacoustic oscillation mode Ma), which have not been reported in previous studies on swirl non-premixed flames. In addition, we comprehensively analyzed the flame pulsation characteristics under the two oscillation modes. Finally, the coupling degrees between velocity fluctuations, fuel pressure fluctuations, and heat release fluctuations were analyzed using the Rayleigh Index (RI), and it was found that in the acoustic-induced vortex mode, a complete feedback loop was not formed between the combustor and the fuel pipeline, which was the main reason for the significant difference in the pressure fluctuation amplitude near 230 Hz and 330 Hz.