The contact process is a simple model for the spread of an infection in a structured population. We consider a variant of this process on Galton-Watson trees, where vertices are equipped with a random fitness representing inhomogeneities among individuals. In this paper, we establish conditions under which the contact process with fitness on Galton-Watson trees exhibits a phase transition. We prove that if the distribution of the product of the offspring and fitness has exponential tails then the survival threshold is strictly positive. Further, we show that, under certain conditions on either the fitness distribution or the offspring distribution, there is no phase transition and the process survives with positive probability for any choice of the infection parameter. A similar dichotomy is known for the contact process on a Galton-Watson tree. However, we see that the introduction of fitness means that we have to take into account the combined effect of fitness and offspring distribution to decide which scenario occurs.