Tree health and growth rate must both be considered in Scots pine breeding for harsh areas such as northern Sweden. Univariate (UV) and multivariate (MV) multi-environment trial (MET) analyses of tree vitality (a measure of tree health) and height (a measure of growth rate) were conducted for four series of open-pollinated Scots pine progeny trials (20 trials total), to evaluate age trends, patterns, and drivers of genotype-by-environment interaction (G × E). The lowest standard errors were obtained for the MV MET analyses, indicating that MVanalyses are preferable to UVanalyses. By incorporating factor-analytic structures, the most complex data sets could be handled, suggesting that factor-analytic analyses are preferred for evaluation of forest progeny trials. We detected strong patterns of G × E for both tree vitality and height, and the driver of G × E was found mainly to be differences in degree day temperature sum, such that G × E was higher between trials with more contrasting temperature sums. The genetic correlations, between vitality and height within sites, were generally positive and were driven by the harshness of the trial; mild trials had lower genetic correlations than did harsh trials. The sign of the across-site genetic correlations between vitality and height changed from positive to negative in some cases, as the differences between the temperature sum of the trials increased. These findings support the hypothesis that tree height assessed in harsh environments with low survival is likely to reflect health and survival ability to a greater extent than growth capacity.