A lot of studies on rotating detonation engines have been carried out due to the higher thermal efficiency. However, the number, rotating directions, and intensities of rotating detonation waves are changeful when the flow rate, equivalence ratio, inflow conditions, and engine schemes vary. The present experimental results showed that the combustion mode of a rotating detonation engine was influenced by the combustor scheme. The annular detonation channel had an outer diameter of 100 mm and an inner diameter of 80 mm. Air and hydrogen were injected into the combustor from 60 cylindrical orifices in a diameter of 2 mm and a circular channel with a width of 2 mm, respectively. When the air mass flow rate was increased by keeping hydrogen flow rate constant, the combustion mode varied. Deflagration and diffusive combustion, multiple counterrotating detonation waves, longitudinal pulsed detonation, and a single rotating detonation wave occurred. Both longitudinal pulsed detonation and a single rotating detonation wave occurred at different times in the same operation. They could change between each other, and the evolution direction depended on the air flow rate. The operations with a single rotating detonation wave occurred at equivalence ratios lower than 0.60, which was helpful for the engine cooling and infrared stealth. The generation mechanism of longitudinal pulsed detonation is developed.