The numerical investigation of the statics and dynamics of systems in non-equilibrium in general, and under shear flow in particular, has become more and more common. However, not all the numerical methods developed to simulate equilibrium systems can be successfully adapted to out-of-equilibrium cases. This is especially true for thermostats. Indeed, even though thermostats developed to work under equilibrium conditions sometimes display good agreement with rheology experiments, their performance rapidly degrades beyond weak dissipation and small shear rates. Here we focus on gauging the relative performances of three thermostats, Langevin, dissipative particle dynamics, and Bussi-Donadio-Parrinello under varying parameters and external conditions. We compare their effectiveness by looking at different observables and clearly demonstrate that choosing the right thermostat (and its parameters) requires a careful evaluation of, at least, temperature, density and velocity profiles. We also show that small modifications of the Langevin and DPD thermostats greatly enhance their performance in a wide range of parameters.