To solve the problem that the channel conditions in asymmetric cooperative FSO communication systems are not fully utilized, and the data reliability deteriorates due to high-order modulation, we proposed a layered modulation, joint physical-layer network coding scheme. In this scheme, we first designate the data priority of the information to be transmitted at the source node. Then, the transmission power of different proportions is allocated to the data based on its priority. Then, the modulated data is sent to each node, and physical-layer network coding is performed on the received data at the relay node. Finally, the relay node sends the encoded information to the destination node, and the destination node recovers the original information using the physical-layer network coding scheme. The simulation results showed that when the average signal-to-noise ratio of the channel was 15 dB, the BER of the cooperative FSO communication system could be reduced to below 10−8. In the strong atmospheric turbulence channel, the cooperative FSO communication system can obtain a signal-to-noise ratio gain of about 1.5 dB. Under strong atmospheric turbulence, this scheme could also improve the average channel capacity performance of a cooperative FSO communication system.

To solve the problems of interruption events and the high bit-error rate of the FSO system caused by atmospheric turbulence, an FSO cooperative communication scheme based on system polarization code is proposed. In this scheme, the upper limit of the average bit-error rate of the atmospheric turbulence channel is used to construct the frozen bit set of the system polarization code, and the information bit set of the S−R link is recovered by using the ownership relationship between the frozen bit set of the S−R link and the S−D link at the destination node. Finally, the information bits of the S−R link and S−D link are combined using equal gain combination, and the original information is recovered by decoding. The simulation results show that the FSO cooperative communication system can overcome the influence of atmospheric turbulence and improve the system performance, and the bit-error rate performance of the FSO cooperative communication system can be improved by at least 0.5dB; the outage probability of the FSO cooperative communication system can be reduced to less than 10−7, and it shows a stable inhibitory effect on strong turbulence conditions.

An adaptive optical wavefront recovery method based on a residual attention network is proposed for the degradation of an Orbital Angular Momentum multiplexing communication system performance caused by atmospheric turbulence in free-space optical communication. To prevent the degeneration phenomenon of neural networks, the residual network is used as the backbone network, and a multi-scale residual hybrid attention network is constructed. Distributed feature extraction by convolutional kernels at different scales is used to enhance the network’s ability to represent light intensity image features. The attention mechanism is used to improve the recognition rate of the network for broken light spot features. The network loss function is designed by combining realistic evaluation indexes so as to obtain Zernike coefficients that match the actual wavefront aberration. Simulation experiments are carried out for different atmospheric turbulence intensity conditions, and the results show that the residual attention network can reconstruct the turbulent phase quickly and accurately. The peaks to valleys of the recovered residual aberrations were between 0.1 and 0.3 rad, and the root means square was between 0.02 and 0.12 rad. The results obtained by the residual attention network are better than those of the conventional network at different SNRs.

An improved non-convex optimized phase recovery algorithm is used to compensate for wavefront aberrations caused by atmospheric turbulence and pointing errors in the vortex beam. The algorithm is divided into two parts: initialization and iteration. To reduce the effect of outliers, truncation rules are formulated in the initialization phase using the robustness of the sample median to obtain an initial value that is close to the global optimum. The relationship between the results of adjacent iterations is used in the iterations to calculate new weight coefficients, which are applied to the gradient descent to ensure the accuracy of the recovery results. Simulation experiments are carried out for different channel environments and different modes, and the results show that the improved phase recovery algorithm can accurately compensate for distorted wave fronts. The improved algorithm recovers the best results at different turbulence intensities and under the influence of different pointing errors. The recovered Strehl ratio can reach 0.9 and the mode purity can reach 0.92. Single-mode and multi-mode simulations were carried out, and the results show that the improved algorithm is effective and robust.