One of the main obstacles faced when decreasing channel spacing in optical wavelength division multiplexing (WDM) systems is the linear crosstalk resulting from spectral overlap between neighboring channels. Methods of reducing channel bandwidth and corresponding crosstalk using optical pre-filtering are reviewed along with the limitations of these techniques. High speed transmitter side digital signal processing (DSP) and digitalto-analog converters enable the use of more powerful techniques that precisely shape and limit the bandwidth of the transmitted optical signal. These techniques allow channel spacing to be reduced to nearly the theoretical minimum Nyquist spacing with only negligible crosstalk penalty. We review well known principles of linear modulation theory and apply them to coherent optical transmission systems to describe the desired channel transmit spectra along with the associated transmit and receive DSP. The implementation issues of a typical optical transmitter, which uses these techniques, are discussed. Experimental results of a laboratory long-haul WDM optical system using spectrally shaped PDM-16-QAM type modulation are presented demonstrating near Nyquist channel spacing on a trans-pacific length system with more than 44 terabits of capacity. We also present an additional application of these spectral shaping techniques, which can carefully expand the bandwidth of the transmitted channels to reduce non-linear transmission penalties. Experimental results show the advantages of these spectral broadening techniques and their ability to trade spectral efficiency for increased reach.