A novel compact multilayer multibeam antenna array based on substrate integrated waveguide (SIW) is proposed in this letter for 5G millimeter wave (mm‐wave) applications. A multimode beam forming network (BFN) cascaded by phase compensation and coupling structures is folded to make the antenna array more compact. The phase difference between the output ports of traditional multimode beamforming network is very unstable, which would seriously affect the multibeam performance of antenna array. Therefore, the compensating phase shifter adopts the SIW transmission line structure with unequal length and width, which enables the multimode BFN achieving better performance in the bandwidth of 27–29 GHz. Four slot arrays are employed as the radiation structure. The overall structure of the proposed antenna consists of three parts, including a multimode beamforming network, a compensated phase shifter, and an SIW slot array. If these components are cascaded directly, the size of the antenna would be very large and the aperture efficiency will be low. So the proposed antenna is folded into three layers, one part for each layer. The overall antenna size is 66.7 mm × 47.5 mm (6.23λ × 4.44λ at 28 GHz). The antenna array is designed, fabricated, and measured. The measured results show that four deflected beams are realized in the band of 27–29 GHz. The radiation beams demonstrate a wide azimuthal coverage of ±48 deg, and a peak gain of 16.8 dBi for Port 1 excitation and 17.3 dBi for Port 2 excitation. The simulated radiation efficiency is better than –2 dB from 27 to 29 GHz. The proposed antenna features a compact size, low profile, good radiation performance, which is well suited for applications in the 5G mm‐wave communication systems.