In this paper we focus on modulation formats for optical transmission networks. The most widely used intensity formats such as Non Return to Zero, Return to Zero, Carrier-Suppressed Return to Zero and duobinary are investigated in terms of bit error rate, Q-factor, optical reach and dense wavelength division multiplexing grid, in order to find out their physical limitations and system performance in a transmission system with given parameters. However, phase-based modulation formats like Differential Phase-Shift Keying and Differential Quadrature Phase-Shift Keying can perform better at the cost of increased transceiver's complexity. Some formats can benefit from polarization division multiplexing to enable higher spectral efficiency, optical reaches, optical to signal noise ratio and chromatic dispersion tolerances. Simulations are performed in OptSim software environment based on the Time Domain Split Step method which uses full band simulation and offers aliasing errors immunity, accurate differential group delays, parallel computing, etc.