In this paper, we focus on the effect of nonlinear high‐power amplifiers (HPA) on the multiple‐input‐multiple‐output space‐time block coded (MIMO‐STBC) systems. In order to compensate the HPA nonlinearity, we propose a new receiver scheme based on a neural network algorithm in conjunction with the maximal‐ratio combining (MRC) technique. The performances of the proposed nonlinear network (NLN), called NLN‐MRC receiver, are evaluated for a MIMO‐STBC systems over uncorrelated Rayleigh fading channels. Analytic expressions of the average symbol error rate and the error vector magnitude are delivered. We also analyse the channel capacity of the considered system assuming the perfect knowledge of the channel coefficients and the use of the water‐filling approach.
Simulation results show that the proposed compensation technique can efficiently reduce the effect of HPA distortions. In addition, we note an excellent agreement between analytic expressions and Monte‐Carlo simulation curves. Furthermore, the proposed adaptive NLN‐MRC scheme has a low complexity, fast convergence, and best performance than its competitors given in the literature. Copyright © 2014 John Wiley & Sons, Ltd.