Ground-state cooling and pure state preparation of a small object that is embedded in a thermal environment is an important challenge and a highly desirable quantum technology. This paper proves, with two different methods, that a fundamental constraint on the cooling dynamic implies that it is impossible to cool, via a unitary system-bath quantum evolution, a system that is embedded in a thermal environment down to its ground state, if the initial state is a factorized product of system and bath states. The latter is a crucial but artificial assumption included in numerous tools that treat system-bath dynamics, such as master equation approaches and Kraus operator based methods. Adopting these approaches to address ground state and even approximate ground state cooling dynamics should therefore be done with caution, considering the fundamental theorem exposed in this work.