To reduce the dimensions of relativistic backward wave oscillators (RBWOs) operating in the low frequency regime of less than 2 GHz, the theory of transversal and longitudinal mode selections are introduced in this paper. The transversal mode selection is achieved using the property of “surface wave” of the coaxial slow-wave structure (SWS) to excite the quasi transverse electromagnetic (quasi-TEM) mode without the higher transverse magnetic (TM) modes and it is proved that the coaxial SWS may decrease the transversal dimension of the SWS sections. In addition, the S-parameter method is employed to investigate the longitudinal resonant characteristic of the finite-length SWS, and the scheme of longitudinal mode selection is put forward. It is proposed that the introduction of a well-designed coaxial extractor to slow-wave devices can help to achieve the longitudinal mode selection and reduce the period number of the SWS, which not only can make the devices more compact, but also can avoid the destructive competition between various longitudinal modes, therefore can enhance the efficiency and stabilize the frequency. To sum up, the physical mechanisms of transversal and longitudinal mode selections ensure that the microwave is produced with a single mode and a narrow band. Based on the above discussion, a compact L-band coaxial RBWO is investigated and optimized in detail with the particle-in-cell KARAT code (V. P. Tarakanov, Berkeley Research Associates, Inc., 1992). In simulation, the L-band coaxial RBWO, driven by a 700 kV, 11 kA electron beam, comes to a nonlinear steady state in 20 ns. High-power microwave of quasi-TEM mode is generated with an average power of 2.66 GW, a frequency of 1.6 GHz, and power conversion efficiency of 34.5% in durations of 30–60 ns.