With the increment of cable length, long stay cables are prone to experience high-mode vortex-induced vibrations (VIVs) at normal wind velocities, and the VIV-prone mode range also becomes wider. Existing dampers cannot supply sufficient damping for such a wide range of high-modes. Inerter-based dampers (IBDs), which take advantage of the two-terminal inertial device dubbed inerter, have been proved to have a better control performance than conventional dampers. However, existing studies on IBDs only focused on the first several cable modes, which cannot cover the wide range of VIV-prone modes of long stay cables. The high-mode and multi-mode VIV control by using IBDs is investigated in this study. The governing equations of the cable-IBD systems under VIV are first established. The control efficiency of the IBDs and the influence of optimum design strategies are compared. The use of two IBDs to further enhance the control efficiency is also discussed. The results show that a three-element IBD in this study is quite effective for high-mode and multi-mode VIV control of long stay cables. Moreover, the multi-mode control efficiency can be further improved through using two IBDs. The results in this study can guide the design of IBDs for VIV control of long stay cables.