In-vacuum undulators have been widely operated in many synchrotron radiation facilities across the world. They usually are required to be operated at a smaller magnet gap than those of other undulators. Thus, operating challenges including impedance effects on the stored electron beam are introduced by these devices. In this paper, we report the efforts in solving the problem of coupled-bunch instabilities caused by an in-vacuum undulator in the SPEAR3 storage ring. Using beam based measurements, cold rf measurements, and numerical simulations, the source of the beam instabilities is characterized as trapped modes in the vacuum chamber. Using numerical models, we explored several approaches to reduce the strength of the trapped modes and found that ferrite dampers were the most effective and simplest way for mode damping in our SPEAR3 in-vacuum undulator. The results of the first rf cold measurement on an invacuum undulator equipped with these ferrite dampers agree well with numerical simulations.