SummaryEfficient spectrum utilization is a promising technique for a prolonged unused radio frequency (RF) spectrum in a wireless network. In this paper, an adaptive spectrum sharing cognitive radio (CR) network has been proposed consisting of a primary user (PU) and secondary user transmitter (SU − Tx) that communicates with secondary user receiver (SU − Rx) via multiantenna‐based proactive decode‐and‐forward (DF) relay selection scheme. In our model, strategically an adaptable joint venture on underlay/overlay protocol is defined based on channel occupancy using spectrum sensing technique. Here, secondary transmitters (i.e., source transmitter) continuously sense the PU activities by energy detector and can simultaneously transmit to secondary receivers. Depending on sensing result secondary transmitters automatically switches in underlay mode if PU is active otherwise operates in overlay mode. The advantage of this scheme is that the joint mode of transmission allows the SUs to maximize their transmission rate. The outage performance at SU − Rx and closed‐form expressions of joint underlay/overlay protocol has been evaluated. The power control policies at different transmitter nodes are taken care of. With the same diversity order, a trade‐off between multiantenna and multirelay is shown. This comparison shows improvement in outage behavior when the count in relays surpasses the number of antennas. Finally, the analytical model of smart efficient spectrum utilization without harming license users in CR is validated by MATLAB simulation.