SummaryIn the realm of 5G mobile wireless communication, cognitive radio (CR) plays an important role in improving radio spectrum efficiency by overcoming challenges such as spectrum shortage versus under‐utilization associated with the traditional fixed frequency assignment. “Interweave CR, underlay CR, and overlay CR” are the three primary paradigmatics in general. The present study focuses on the overlay CR paradigm, wherein secondary users (SUs) have all been expected to be provided with superior “encoding and signal processing techniques” to improve primary users' (PUs') communication. Furthermore, to address concerns associated with inefficient spectrum usage, multiplexing methods like “non‐orthogonal multiple access (NOMA) and spatial modulation (SM)” have been implemented. NOMA makes use of the power domain to maximize spectrum usage, whereas SM makes use of the spatial domain. Multiple antennas, on the other hand, are expensive in terms of energy, magnitude, and equipment. Antenna selection is a low‐cost, low‐complexity method of capturing many of the NOMA system's advantages. As a consequence, this study offers the fitness familiarized lion algorithm (FFLA), a novel methodology for selecting the best antennas. Furthermore, the selection procedure involves identifying the stated objectives, which include energy usage and error minimization. Indeed, the suggested technique is an enhanced version of the lion algorithm (LA). Ultimately, the suggested work's performance is compared to and proven against other traditional models.