We propose a novel cooling scheme for realising single photon sideband cooling on particles trapped in a state-dependent optical potential. We develop a master rate equation from an ab-initio model and find that in experimentally feasible conditions it is possible to drastically reduce the average occupation number of the vibrational levels by applying a frequency sweep on the cooling laser that sequentially cools all the motional states. Notably, this cooling scheme works also when a particle experiences a deeper trap in its internal ground state than in its excited state, a condition for which conventional single photon sideband cooling does not work. In our analysis, we consider two cases: a two-level particle confined in an optical tweezer and Li atoms confined in an optical lattice, and find conditions for efficient cooling in both cases. The results from the model are confirmed by a full quantum Monte Carlo simulation of the system Hamiltonian. Our findings provide an alternative cooling scheme that can be applied in principle to any particle, e.g. atoms, molecules or ions, confined in a state-dependent optical potential.