Summary
The coexistence of geostationary satellite orbit (GSO) and non‐GSO (NGSO) fixed satellite service systems within the same spectrum to enhance the spectrum efficiency in Ka band has attracted many interests lately. However, the coexistence of GSO‐NGSO satellite systems with limited shared spectrum can cause in‐line interference from a satellite to other satellite's earth terminals. In this context, this contribution investigates three possible power control approaches to mitigate the in‐line interference caused by an NGSO satellite to the GSO earth terminal, while the NGSO satellite is crossing the GSO satellite's illumination zone. Moreover, three types of interference mitigation techniques, namely, range‐based, traffic‐aware, and cognitive power control techniques are investigated. Furthermore, to increase the spectrum efficiency of the GSO‐NGSO fixed satellite service terrestrial network, we formulate and solve an optimization problem with the objective of minimizing the inter‐site distance between earth user‐terminals. More specifically, we find the minimum distance from a GSO earth terminal, where an NGSO earth terminal can be implemented subject to minimizing received interference level at the victim system. Finally, our comparative results show that the cognitive power control technique performs the best in protecting the victim receiver from the in‐line interference. Copyright © 2016 John Wiley & Sons, Ltd.