In this article, we successfully validated the VSTF on coherent optical ultradense WDM and analyzed the transmission impact of XPM/FWM on 16-QAM signals. From the studies it is visible that for mixed phase/amplitude coherent signals, in the higher power symbols we will have similar degradation caused by XPM and FWM, and it becomes more visible for longer fibers.|| ACKNOWLEDGMENTS This work was supported by the EURO-FOS project, a Network of Excellence funded by the EU 7th ICT-FP. J.D. Reis also acknowledges his PhD grant from FCT (SFRH/43941/2008). 2. X. Zhou et al., 64-Tb/s (640X107-Gb/s) PDM-36QAM Transmission over 320km using both pre-and post-transmission digital equalization, In: Proceedings of Conference Optical Fiber Communication (OFC), 2010, p.PDPB9. 3. K.V. Peddanarappagari and B.M. Pearce, Volterra series transfer function of single-mode fibers, IEEE/OSA J Lightwave Technol 15 (1997), 2232-2241. 4. R. Weidenfeld, M. Nazarathy, R. Noe, and I. Shpatzer, Volterra nonlinear compensation of 112 Gb/s ultra-long-haul coherent optical OFDM based on frequency-shaped decision feedback, In: