In a long-haul optical fiber communication system, fiber attenuation, dispersion, and nonlinearity combine with non-deterministic noise from optical amplifiers used for periodic regeneration and cause adverse effects on system performance. In this dissertation, we study and mitigate such undesirable effects.We present a modified nonlinear decision feedback equalizer designed for use in a legacy optical communication system with periodic dispersion compensation.The effects of noise and nonlinearity on the equalizer coefficients are investigated, and a suboptimal convergence algorithm to reduce such effects is proposed and verified.Noting the limited ability of existing signal processing tools to combat signalnoise nonlinear interaction effects, we next consider a fundamental scenario to study these effects. We apply Gaussian mixture modeling (GMM) techniques to better understand how noise interacts with the signal in a nonlinear optical fiber span. We validate our technique and learn that at higher levels of nonlinearity, the GMM analysis is more accurate than an additive Gaussian noise or a Volterra series transfer function model.Finally we present an approach to generalize our analysis. We validate our claims that transmitting a small number of pulses is a good approach to predict the analysis of a practical communication system. We also show that using a 3-or higher-order GMM is necessary to fully understand the nonlinear interaction.