The millimeter wave band is becoming popular for mobile broadband usage such as fifthgeneration (5G) mobile and mobile satellite services utilizing earth stations in motion (ESIM). According to the 2019 World Radiocommunication Conference (WRC-19), the 5G and ESIM systems will operate in adjacent frequency bands bounded by 27.5 GHz; therefore, the adjacent channel compatibility between ESIM and 5G should be verified. Both, the minimum coupling loss (MCL) and Monte-Carlo (MC) methods are applied to assess the worst and most practical interference effects, respectively, for all types of ESIM including the following: maritime ESIM (M-ESIM), land ESIM (L-ESIM), and aeronautical ESIM (A-ESIM). The distance and guard band between the two systems are indicated by the compatibility conditions. In addition to the conventional interference-to-noise ratio (I/N), the throughput loss of a 5G system is proposed to assess the performance degradation caused by the ESIM interference. Two orthogonal frequency division multiplexing (OFDM) waveforms are proposed to suppress ESIM power leakage into an adjacent channel. A mathematical expression regarding the power spectral density (PSD) and frequency dependent rejection (FDR) is derived for these waveforms, suggesting that the interference can be alleviated. A measured single carrier waveform of a commercial ESIM equipment is used as the benchmark against the proposed OFDM waveforms. The windowed OFDM is able to reduce the guard band by 50-77%. The results obtained for various elevation angles of the ESIM antenna are determined to be applicable to various regions globally. INDEX TERMS Adjacent channel compatibility, spectral coexistence study, fifth-generation (5G) cellular mobile communications, earth station in motion (ESIM), fixed satellite service (FSS), orthogonal frequency division multiplexing (OFDM), CP-OFDM, windowed OFDM, power spectral density, frequency dependent rejection, minimum coupling loss (MCL), Monte-Carlo (MC).