Topological insulator materials, including Bi2Te3, Sb2Te3, Sb2Te3 and Bi2Se3, have attracted some attention due to their narrow band gap, high carrier mobility, wide spectral absorption range and other characteristics. Compared with the traditional topological insulator composed of two elements, the new multi-compound topological insulator material can integrate the physical advantages of each element, which is helpful to build an experimental platform with rich physical properties. In this work, the nonlinear optical characteristics of the quaternary material BiSbTeSe2 is reported in the erbium-doped fiber laser. Using the BiSbTeSe2 as a saturable absorber material, the passive Q-switched and mode-locked fiber lasers are achieved. The pulse duration and SNR of the Q-switched fiber laser are 854 ns and 70 dB, respectively. Meanwhile, the pulse duration and SNR of the mode-locked fiber laser are 259 fs and 87.75 dB, respectively. This work proves that the BiSbTeSe2 has a considerable application prospect as a saturable absorber in fiber lasers, and provides a new reference for the selection of high-performance saturable absorber materials.