5G mobile networks targets wireless connection capacity up to 10 Gb/s. For this purpose, we propose a method to considerably increase capacity. In this paper first, we show how to compensate the effects of polarization mode dispersion (PMD) in systems with double polarizations where PMD in such systems could cause fluctuations in optical transmission due to crosstalk and cross phase modulation. Second, we show how to enhance system capacity benefiting from polarization multiplexing (POL-MUX) technique which can provide double bandwidth efficiency. Based on the simulation results, we have achieved optimum system performance and we were able to reduce the PMD effect using pre- and post-compensation. We also have improved the POL-MUX technique using coherent detection in case of 16/64 QAM modulations. The results were achieved by implementing polarization controllers, polarization beam combiners and splitters, as well as polarization phase shifters.