New families of time-dependent potentials related with the stationary singular oscillator are introduced. This is achieved after noticing that a non stationary quantum invariant can be constructed for the singular oscillator. Such invariant depends on coefficients that are related to solutions of an Ermakov equation, the latter becomes essential since it guarantees the regularity of the solutions at each time. In this form, after applying the factorization method to the quantum invariant, rather than the Hamiltonian, one manages to introduce the time parameter into the transformation, leading to factorized operators which are the constants of motion of the new time-dependent potentials. Under the appropriate limit, the initial quantum invariant reduces to the stationary singular oscillator Hamiltonian, in such case, one recovers the families of potentials obtained through the conventional factorization method and previously reported in the literature. In addition, some special limits are discussed such that the singular barrier of the potential vanishes, leading to non-singular time-dependent potentials.