The proliferation of biological data with large numbers of samples and many dimensions is kindling hope that life scientists will be able to fit statistical and machine learning models that are highly predictive and interpretable. However, large biological data sets are commonly burdened with an inherent trade-off: in-sample prediction will improve as additional predictors are included in the model, but this may come at the cost of poor predictive accuracy and limited generalizability for future or unsampled observations (i.e., out-of-sample prediction). To confront this problem of overfitting, sparse modeling methods can correctly place low weight on unimportant predictor variables, thereby narrowing in on the key predictors associated with variation in the response. We competed nine methods to quantify their performance in variable selection, estimation, and prediction using simulated data sets with different sample sizes, numbers of predictors, and strengths of effects. Overfitting was typical for many methods and all simulation scenarios, as demonstrated by large in-sample R2 and low out-of-sample R2. Despite this, in-sample and out-of-sample prediction converged on the true predictive target for simulations with more observations, larger effects of causal variables, and fewer predictors. Analyses of simulations showed that accurate variable selection to support process-based understanding will be unattainable for many realistic sampling schemes. We use our analyses to characterize data attributes in which statistical learning is possible, and illustrate how some sparse methods can be used to achieve predictive accuracy while mitigating and learning the extent of potential overfitting.