Nonlinear phase noise in a fiber optical communication system with 16 Quadrature Amplitude Modulation (16-QAM) has been analyzed in this work. Three new constellation schemes are compared with the post compensation using mean shift rotation method. The new constellations focus on optimizing the ratio between the amplitudes in the 16-QAM and placing the points of 16-QAM in different patterns to minimize the effect of the nonlinear phase noise. In previous works, the best performance was achieved with circular constellation without any additional compensation techniques. The performance is improved by improved circular constellation with optimized radii, five shells circular constellation and improved five-shell circular constellation in this work. The improved five-shell circular constellation diagram improves the bit error rate performance by one order of magnitude from the existing circular constellation at the transmission distance of 3000 km. This result is due to the fact that moderate amplitude signals are more tolerance to nonlinear phase noise.Index Terms-16-QAM, Constellation diagram, nonlinear phase noise, optical fiber communication.