Atmospheric turbulence causes signal beam wavefront distortion at the receiving end of a coherent detection system, which decreases the system mixing efficiency. Based on the coherent detection theory, this study establishes a mathematical model of wavefront distortion with mixing efficiency and mixing gain. It also analyzes the improvement limits of wavefront correction on mixing efficiency and mixing gain under different atmospheric turbulence intensities and experimentally measures them. Simulation results show that the mixing efficiency can be improved to 51%, 55%, and 60% after correcting for tilt, defocus, and astigmatism terms, respectively, when turbulence intensity D/r0 is 2. The mixing gain with homodyne detection is 3 dB higher than heterodyne detection. Meanwhile, the wavefront correction orders required for optimal mixing efficiency are higher than the heterodyne correction order. In the experiment, Haso4 NIR + DM 40 was used, and the turbulence intensity D/r0 was 2. After the closed-loop control algorithm corrects the tilt, defocus, and astigmatism terms, the indoor experimental results showed that the mixing efficiency is improved to 36%, 47%, and 62%, respectively. The outdoor experimental results showed that the mixing efficiency improved to 36%, 51%, and 68%, respectively.