SummaryMassive multiple‐input multiple‐output orthogonal frequency division multiplexing (MIMO‐OFDM) system is an approach to increase data rate and spectral efficiency of 5G wireless communication systems. It's easy to analyze the performance of any system with the assumption of immobile users, but if the user is moving with varying velocity, then the communication between user and base station (BS) is very difficult to maintain. Under high mobility, rapidly fading, doubly dispersive, and nonstationary environment, the conventional system fails to respond and leads to the problem of intercarrier interference (ICI). To overcome these problems, fractional Fourier transform (FrFT) has been used in place of Fourier transform (FT) to increase the correlation coefficient, to mitigate effect of ICI, and to improve bit error rate. In this paper, novel formulae for correlation coefficient, average power of ICI, and closed form bit error expression have been derived for the proposed system by considering fast‐fading model and imperfect channel state information. Finally, the comparison of both the systems conventional and proposed proved that the proposed system has improved bit error rate by 40% and intercarrier interference cancelation by 70%.