In face of the small-scale structure problems of the collisionless cold dark matter (DM) paradigm, a popular remedy is to introduce a strong DM self-interaction which can be generated nonperturbatively by a MeV-scale light mediator. However, if such the mediator is unstable and decays into SM particles, the model is severely constrained by the DM direct and indirect detection experiments. In the present paper, we study a model of a self-interacting fermionic DM, endowed with a light stable scalar mediator. In this model, the DM relic abundance is dominated by the fermionic DM particle which is generated mainly via the freeze-out of its annihilations to the stable mediator. Since this channel is invisible, the DM indirect detection constraints should be greatly relaxed. Furthermore, the direct detection signals are suppressed to an unobservable level since fermionic DM scatterings with a nucleon appear at one-loop level. By further studying the bounds from the CMB and BBN on the visible channels involving the dark sector, we show that there is a large parameter space which can generate appropriate DM self-interactions at dwarf galaxy scales, while remaining compatible with other experimental constraints.