Fifth generation (5G) technology aims to provide high peak data rates, increased bandwidth, and supports a 1 millisecond roundtrip latency at millimeter wave (mmWave). However, higher frequency bands in mmWave comes with challenges including poor propagation characteristics and lossy structure. The beamforming Butler matrix (BM) is an alternative design intended to overcome these limitations by controlling the phase and amplitude of the signal, which reduces the path loss and penetration losses. At the mmWave, the wavelength becomes smaller, and the BM planar structure is intricate and faces issues of insertion losses and size due to the complexity. To address these issues, a dual-layer substrate is connected through the via, and the hybrids are arranged side by side. The dual-layer structure circumvents the crossover elements, while the strip line, hybrids, and via-hole are carefully designed on each BM element. The internal design of BM features a compact size and low-profile structure, with dimensions of 23.26 mm × 28.92 mm (2.17 λ0 × 2.69 λ0), which is ideally suited for the 5G mmWave communication system. The designed BM measured results show return losses, Sii and Sjj, of less than −10 dB, transmission amplitude of −8 ± 2 dB, and an acceptable range of output phase at 28 GHz.