High bit rate modulation is an essential aspect of next-generation WiFi and plays a crucial role in achieving high data rates and increased network capacity. high-bit rate modulation uses advanced modulation schemes, such as 4096-QAM, to increase the maximum number of bits sent over a particular frequency band. Using nonlinear amplifiers in wireless communication systems will significantly improve the performance of high bit rate modulation techniques, such as 4096-QAM. Nonlinear amplifiers can cause a range of impairments, including intermodulation, harmonic distortion, and amplitude compression, which can affect the transmitted signal quality. It is essential to minimize the effects of these impairments through proper design and optimization of the amplifier to ensure the highest quality possible of the transmitted signal. In this paper, M-QAM with nonlinear amplifier system design and implanted with MATLAB Simulink where in terms of Transmitted and Received spectrum signal and constellation diagram and calculated error vector magnitude (EVM) and modulation error ratio (MER) measurements for 1024, 2048, and 4096-QAM that used in WiFi 6, WiFi 6E, WiFi 7, respectively.