This paper provides an overview of latest progress on the novel advanced digital signal processing (DSP) techniques for long-haul mode division multiplexing (MDM) systems with high capacity. Space-division multiplexing (SDM) techniques have been developed for a period to increase the capacity of optical communication system by at least one order of magnitude through MDM techniques using few-mode fibers (FMFs) or multi-core multiplexing (MCM) using multi-core fibers (MCFs). The signals in MDM links are mainly impaired by the linear and nonlinear effects in FMFs, making DSP techniques become necessary to undo these impairments. In this paper, we not only review the advanced multiple-input multiple-output (MIMO) DSP techniques for compensating linear impairments in FMFs, but also enclose the state of the art of novel DSP techniques to deal with nonlinear effects. Firstly, we introduce the MIMO schemes for equalizing modal crosstalk and modal dispersion. Then, we focus on the fast tracking of time-varying (TV) channels in FMF links through frequency-domain (FD) recursive least square (RLS) algorithm. Besides, we also cover the mainstream DSP solutions for mode-dependent loss (MDL) and several possible methods to compensate nonlinearity in FMF. Moreover, artificial intelligence (AI) technologies are also discussed for its high nonlinearity tolerance and may bring a revolution in MDM systems on the process of channel equalization, link monitoring, etc. In the end, a brief conclusion and perspective will be provided.