The problem of automatic modulation format identification (MFI) is one of the main challenges in adaptive optical systems. In this work, we investigate MFI in superchannel optical networks. The investigation is conducted by considering the classification of seven multiplexed channels of 20 Gbaud, each with six commonly used modulation formats, including polarization division multiplexing (PDM)-BPSK, PDM-QPSK, and PDM-MQAM with (M=8,16,32,64). The classification performance is assessed under different values of optical signalto-noise ratio (OSNR) and in the presence of channel interference, channel chromatic dispersion, phase noise, and 1 st polarization mode dispersion (PMD). Furthermore, the effect of fiber nonlinearity on the MFI accuracy is investigated. A well-established machine learning algorithm based on histogram features and a convolutional neural network has been used in this investigation. Results indicate that accurate identification accuracy can be achieved within the OSNR range of practical systems and that the MFI accuracy of side subcarriers outperforms that of middle subcarriers at a fixed value of OSNR. The results also show that the MFI accuracy of PDM-16QAM and PDM-64QAM are affected more by channel interference than the other modulation formats, especially when the ratio of the subcarrier bandwidth to subcarriers spacing is ≥ 1.4. Finally, laboratory experiments have been conducted for validation purposes. The experimental results were found in good agreement with those achieved by simulation.