Recently, automatic modulation classification (AMC) has acquired a lot of interest in the optical communication community. Most optical wireless communication systems are intended to transmit multimedia content, especially video and speech signals. The optical wireless communication channel has variable characteristics. Hence, there is a need for an adaptive modulation scheme to cope with the varying channel characteristics. Adaptive modulation requires the implementation of adaptive modulation classification at the receiver end. Instead of using complex classification with deep learning techniques, a simple proposed scheme for AMC is introduced in this paper. This proposed scheme is based on a chaotic Baker map (CBM), wavelet image fusion, and autocorrelation estimation. It depends on constellation diagrams for eight modulation formats, including (B/Q/8/16 PSK), (8/16/32/64 QAM). The constellation diagrams are acquired and utilized through the CBM, and they are merged using the wavelet image fusion and stored as reference templates in the system database. After that, the classification of each modulation format depends on estimated correlation scores and a thresholding strategy. Simulation results prove good classification and recognition for all studied modulation formats.

Transfer learning appears to be a potential method for transferring information from general to specialized activities. Unfortunately, experimenting using various transfer learning models does not yield good results. In this paper, we propose the utilization of the Hough transform (HT) to improve modulation format recognition. HT is utilized to estimate points on constellation diagrams patterns with perspective projection. The HT algorithm extracts the raw image feature lines. The feature edges are then translated into Hough space, where line segments are found. The Hough transform is used to project a constellation diagram in another location and extract features from it. Constellation diagrams for eight different modulation formats (2/4/8/16 - PSK and 8/16/32/64 - QAM) are obtained at optical signal-to-noise ratios (OSNRs) ranging from 5 to 30 dB. This model is based on classification, and transfer learning. The obtained results indicate that, even at low OSNR values, the suggested system is capable of blindly recognizing the wireless optical modulation format with a classification accuracy of up to 99%.