In this paper, we evaluate the selective combination of algorithms for fiber nonlinearity compensation, transceiver impairments mitigation and variable-rate coding in unrepeatered optical transmission. A post-emphasis filter and a two-stage 4×4 multiple-input and multiple-output equalizer compensate the impact of bandwidth limitations and in-phase and quadrature skew. Nonlinear compensation is accomplished by a fast-converging adaptive digital back-propagation algorithm and maximum likelihood sequence estimation. Forward error correction is provided by a variable-rate spatially-coupled low-density parity-check code. The performance and complexity of the proposed digital subsystems are experimentally evaluated by the unrepeatered wavelength division multiplexing transmission of 17×200-Gb/s (DP-16QAM 32-GBd) channels over 350-km of large effective area and low-loss single-mode fibers. The experimental results for different combinations of algorithms elucidate the trade-off of complexity and performance in unrepeatered optical transmission.