The roughness of vapor-deposited thin films can display a nonmonotonic dependence on film thickness, if the smoothening of the small-scale features of the substrate dominates over growth-induced roughening in the early stage of evolution. We present a detailed analysis of this phenomenon in the framework of the continuum theory of unstable homoepitaxy. Using the spherical approximation of phase ordering kinetics, the effect of nonlinearities and noise can be treated explicitly. The substrate roughness is characterized by the dimensionless parameter Q = W0/(k0a 2 ), where W0 denotes the roughness amplitude, k0 is the small scale cutoff wavenumber of the roughness spectrum, and a is the lattice constant. Depending on Q, the diffusion length lD and the Ehrlich-Schwoebel length lES, five regimes are identified in which the position of the roughness minimum is determined by different physical mechanisms. The analytic estimates are compared by numerical simulations of the full nonlinear evolution equation.