Due to the rapid development of wireless communication technologies, the number of wireless users are radically increasing. Currently, ∼23 billion wireless devices are connected to the internet, and these numbers are expected to increase manifolds in the years to come. The technology growth of the fifth-generation (5G) wireless systems will be needed to meet this high demand of the network. 5G wireless systems offer data-rates of up to 10Gbps, 1-ms latency, and reduced power consumption. It is a known fact that 5G wireless systems will be exploiting beyond the presently used 3 GHz microwave and millimetre-wave (mm-wave) frequency bands. This is the primary driver in the development of the 5G wireless system. Multi-beam Phased array antenna (PAA) systems are typically used in the deployment of 5G systems for high-gain and directionality. In current 5G and future Beyond 5G (B5G) antenna array systems, beamforming networks (BFNs) such as the Butler Matrix (BM) will play a key role in achieving multi-beam characteristics. So, this paper presents an extensive review of the BM based BFNs, and discusses which type of BM will be suitable for the phased array antenna (PAA) systems in the upcoming 5G and next-generation of B5G wireless systems. Moreover, this paper also summarizes the different types of BM designs based on the number of layers. The BMs are classified into the bi-layer, tri-layer, and four-layer structures. It includes different techniques that have been used to solve the problem of crossing, narrow bandwidth, and size reduction of the BM. From the previous studies, it is found that most of the past research work was performed using the bi-layer BM system, whereas the difficult geometries like tri-and four-layer BM are avoided due to their complex fabrication process. It is also found in this paper that the metamaterial (MTM) based bi-layer BM achieves low insertion-loss and phase-error, excellent bandwidth and compact size, and good S-parameter performance, which makes them an ideal BFN candidate for the upcoming 5G and next-generation B5G systems.INDEX TERMS Butler matrix (BM), metamaterial (MTM), 5G, beyond 5G (B5G), beamforming network (BFN), phased array antenna (PAA) systems.