Broadband nonlinear optical materials towards the deep mid-infrared regime are highly required in medical diagnostics, environmental pollution monitoring, etc. Here, we report the ultra-broadband nonlinear optical absorption behavior of the layered vanadium diselenide (VSe2) towards deep mid-infrared regime experimentally. The layered VSe2 prepared by the liquid-phase exfoliation method exhibits broadband and wavelength-dependent nonlinear saturable absorption and a large modulation depth of over 8% in the 3-10 µm spectral range. By utilizing the layered VSe2 for optical modulation, stable nanosecond pulse with a low Q-switched threshold has been delivered successfully from an erbium-doped ZBLAN fiber laser. The experimental findings can deepen the understanding of the nonlinear optical response of the transition metal dichalcogenides towards the deep mid-infrared regime, and may make inroads for the advancement of mid-IR optoelectronic devices.