A support vector machine (SVM)-based data detection is proposed for coherent optical fiber amplitude phase-shift keying (APSK) communication systems where the nonlinear phase noise is the main system impairment. The performances of the detection with SVMs are investigated for three different 16-APSK modulation formats. In addition, three transmission scenarios with dispersion being considered or not are adopted to simulate and analyze the performances. Compared with the traditional two-stage maximum-likelihood detection, the SVM conducts detection without the need to know the information of transmission link, and it gains a relatively large improvement in the nonlinear system tolerance, particularly in the high nonlinear regime. Compared with quadrature amplitude modulation (QAM), the 16-APSK system can increase the nonlinear system tolerance by 4.88 dB at BER ¼ 1E À 3.Index Terms: Support vector machine (SVM), amplitude phase-shift keying (APSK), nonlinear phase noise (NLPN).