Futuristic mobile data networks are expected to reach higher data rates per user to accommodate the specifications of emerging services, such as triple play and mobile applications. Researchers are carrying out their works to tackle the drawbacks of traditional orthogonal frequency-division multiplexing by designing new waveforms with high spectral efficiency and low out-of-band emissions. Among many others, filter bank multicarrier (FBMC) is an appealing candidate for beyond 5G that fulfills these constraints. Consequently, it is worth to shed light on the performance of FBMC in multichannel transmission system. The main contribution of our paper is to compensate fiber nonlinearity including self-phase modulation and cross-phase modulation effects using adaptive Volterra equalizer in longhaul FBMC system based on a 6 × 20 Gb∕s in wavelength-division multiplexing scheme over 100 km × 30 spans of singlemode fiber (SMF). The impacts of channel spacing, bit rate, and fiber types on system performance are addressed. For a target of bit error rate ¼ 10 −3 , the simulations show that this performance can be reached at −12 dBm of launch power when 5-taps third-order Volterra (TOV) equalizer is used for 12.5 GHz of channel spacing and an improvement of approximately 1 dB is obtained for 7 taps compared with the same equalizer with 3 taps. To further enhance the system performance, standard-SMF is replaced by a new class fiber called pure-silica-core fiber with large effective area, which exhibits a gain of almost 1.3 and 1 dB for 3 and 5 taps, respectively. The complexity burden of the TOV filter is also discussed. © The Authors. Published by SPIE under a Creative Commons Attribution 4.0 International License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI.