Linear signal processing algorithms are effective in dealing with linear transmission channel and linear signal detection, while the nonlinear signal processing algorithms, from the machine learning community, are effective in dealing with nonlinear transmission channel and nonlinear signal detection. In this paper, a brief overview of the various machine learning methods and their application in optical communication is presented and discussed. Moreover, supervised machine learning methods, such as neural networks and support vector machine, are experimentally demonstrated for in-band optical signal to noise ratio (OSNR) estimation and modulation format classification, respectively. The proposed methods accurately evaluate optical signals employing up to 64 quadrature amplitude modulation (QAM), at 32 Gbaud, using only directly-detected data.