Internal Validation of Predictive Logistic Regression Models for Decision‐Making in Wildlife Management

Abstract

Predictive logistic regression models are commonly used to make informed decisions related to wildlife management and conservation, such as predicting favourable wildlife habitat for land conservation objectives and predicting vital rates for use in population models. Frequently, models are developed for use in the same population from which sample data were obtained, and thus, they are intended for internal use within the same population. Before predictions from logistic regression models are used to make management decisions, predictive ability should be validated. We describe a process for conducting an internal model validation, and we illustrate the process of internal validation using logistic regression models for predicting the number of successfully breeding wolf packs in six areas in the US northern Rocky Mountains. We start by defining the major components of accuracy for binary predictions as calibration and discrimination, and we describe methods for quantifying the calibration and discrimination abilities of a logistic regression model. We also describe methods for correcting problems of calibration and future predictive accuracy in a logistic regression model. We then show how bootstrap simulations can be used to obtain unbiased estimates of prediction accuracy when models are calibrated and evaluated within the same population from which they were developed. We also show how bootstrapping can be used to assess coverage rates and recalibrate the endpoints of confidence intervals for predictions from a logistic regression model, to achieve nominal coverage rates. Using the data on successfully breeding wolf packs in the northern Rocky Mountains, we validate that predictions from a model developed with data specific to each of six analysis areas are better calibrated to each population than a global model developed using all data simultaneously. We then use shrinkage of model coefficients to improve calibration and future predictive accuracy for the area-specific model, and recalibrate confidence interval endpoints to provide better coverage properties. Following this validation, managers can be confident that logistic regression predictions will be reliable in this situation, and thus that management decisions will be based on accurate predictions.

A national benchmark investigation of return-to-work outcome rates between African American, Native American or Alaskan Native, Latino, Asian American or Pacific Islander, and Non-Latino White veterans served by state vocational rehabilitation agencies: Application of bootstrap data expansion

“…The association between select characteristics (i.e., race/ethnicity, gender, education level at closure) was tested using multinomial logistic regression and applied bootstrap resample techniques to increase the efficiency of interval validation procedures (Gude et al, 2009;Steyerberg et al, 2001). The procedures are reflected broadly in Fig.…”

Predictive Models for Mutation Carriers in Brugada Syndrome Screening

“…Both issues, of course, lead to unstable coefficient regression estimates [8], [14]. The reliability of a prognostic model constructed in such circumstances should then be evaluated, using adequate procedures that can yield reliable measures of predictive accuracy [6], [23]. It is well known that the apparent accuracy of a model is optimistic, that is, the accuracy obtained when evaluating a model with the same data set that was used to estimate it, will yield an optimistic evaluation [3], [5], [8].…”

“…Furthermore, we use shrinkage methods to improve calibration and future predictive accuracy of the models. Shrinkage techniques are well known tools to improve predictive models in situations like the two referred above: small sample size and correlated predictive variables [6], [8], [21], [22], [25]. Shrinkage is related to overfitting.…”

“…It is well known that the apparent accuracy of a model is optimistic, that is, the accuracy obtained when evaluating a model with the same data set that was used to estimate it, will yield an optimistic evaluation [3], [5], [8]. To overcome this weakness, several procedures for internal validation have been proposed and investigated in the literature [2], [6], [8], [23]. In Steyerberg et al [23], we can find a comparison of some internal validation methods for logistic regression models, using random samples of different sizes drawn from a large data set of clinical records.…”

“…Bootstrapping has been frequently applied to validate logistic regression predictive models in life sciences (e.g. [2], [6], [13], [17], [25]). …”