In coherent optical communication systems, where multiple modulation formats are mixed and variable, the correct identification of signal modulation formats provides the foundation for subsequent performance improvement using digital algorithms. A modulation format identification (MFI) scheme based on signal constellation diagrams and support vector machine (SVM) is proposed. Firstly, the signal constellation diagrams are divided by the fractal dimension of the weighted linear least squares (WLS-FD) algorithm, and the fractal dimension (FD) in each region is calculated, which is regarded as one of the image features. Then, the feature values of the image in different directions are extracted by the gray-level co-occurrence matrix (GLCM), and their mean and variance are calculated, which is regarded as another feature. Finally, the two features are input into the modulation format classifier constructed by the SVM to achieve MFI in coherent optical communication systems. To verify the feasibility and superiority of the scheme, we compare it with the MFI scheme based on higher-order statistical (HOS) features, GLCM features, and FD features, respectively. Further, we built a 30 GBaud coherent optical communication system with fiber lengths of 80 km and 120 km, where the optical signal-to-noise ratio (OSNR) ranges from 0 dB to 30 dB. The proposed MFI scheme identifies seven modulation formats: QPSK, 8QAM, 16QAM, 32QAM, 64QAM, 128QAM, and 256QAM. The results show that compared with the other three schemes, our proposed scheme has a better identification accuracy at low OSNR. In addition, the identification accuracy of this scheme can reach 100% when the OSNR ≥ 10 dB.