In this paper, we investigate the outage probability (OP) performance of amplify-and-forward (AF) cognitive hybrid satellite-terrestrial overlay networks (CHSTONs) with the non-orthogonal multiple access (NOMA) scheme, in which half-duplexing terrestrial secondary networks cooperate with a primary satellite network for dynamic spectrum access. In order to improve the fairness of overlay paradigm, a NOMA-based power allocation profile is determined by instantaneous channel conditions. Considering the generalized shadowed-Rician fading for satellite links and Nakagami-m fading for terrestrial links, we derive the closed-form OP expressions for both the primary and secondary users. Then, the asymptotic OP expressions at the high signal-to-noise ratio (SNR) regime are also obtained to evaluate the achievable diversity order and coding gain. Finally, the numerical simulations are provided to validate the theoretical results as well as the superiority of the NOMA scheme in CHSTONs and proclaim the effect of key parameters on the performance of the NOMA users, such as fading configurations and the power split factor.INDEX TERMS Non-orthogonal multiple access (NOMA), cognitive hybrid satellite-terrestrial overlay networks (CHSTONs), outage probability.

In this paper, the performance of non-orthogonal multiple access (NOMA) based cooperative spectrum sharing in hybrid satellite-terrestrial networks (HSTNs) is investigated, where the primary satellite network recruits the secondary terrestrial network as a cooperative relay. To improve the fairness and spectrum utilization under cooperative spectrum sharing (i.e. overlay paradigm of cognitive radio), the NOMA power allocation profile is determined by instantaneous channel conditioning at the second temporal phase. The closed-form outage probability and approximated ergodic capacity expressions for the primary user (PU) and the secondary user (SU) are derived by decode-and-forward (DF) relay protocols, where the generalized Shadowed-Rician fading and Nakagami-m fading are considered for satellite links and terrestrial links, respectively. Simulation results are conducted for validation of the theoretical derivation and analysis of the impact of key parameters, and prove the superiority of NOMA comparing to conventional orthogonal multiple access (OMA) schemes on cooperative spectrum sharing in HSTNs. Besides, the fairness analysis between the PU and the SU is introduced by Jain's fairness index (JFI). INDEX TERMS Non-orthogonal multiple access (NOMA), hybrid satellite-terrestrial networks (HSTNs), cooperative spectrum sharing, outage probability (OP), ergodic capacity.