We compared the performance of linear (GBLUP, BayesB and elastic net) methods to a non-parametric tree-based ensemble (gradient boosting machine—GBM) method for genomic prediction of complex traits in mice. The dataset used contained genotypes for 50,112 SNP markers and phenotypes for 835 animals from 6 generations. Traits analyzed were bone mineral density, body weight at 10, 15 and 20 weeks, fat percentage, circulating cholesterol, glucose, insulin, triglycerides, and urine creatinine. The youngest generation was used as validation subset, and predictions were based on all older generations. Model performance was evaluated by comparing predictions for animals in the validation subset against their adjusted phenotypes. Linear models outperformed GBM for seven out of ten traits. For bone mineral density, cholesterol and glucose, the GBM model showed better prediction accuracy and lower relative root mean squared error than the linear models. Interestingly, for these three traits there is evidence of a relevant portion of phenotypic variance being explained by epistatic effects. Using a subset of top markers selected from a GBM model helped for some of the traits to improve accuracy of prediction when these were fitted into linear and GBM models. Our results indicate that GBM is more strongly affected by data size and decreased connectedness between reference and validation sets than the linear models. Although the linear models outperformed GBM for the polygenic traits, our results suggest that GBM is a competitive method to predict complex traits with assumed epistatic effects.