Is your ad hoc model selection strategy affecting your multimodel inference?

Abstract

Ecologists routinely fit complex models with multiple parameters of interest, where hundreds or more competing models are plausible. To limit the number of fitted models, ecologists often define a model selection strategy composed of a series of stages in which certain features of a model are compared while other features are held constant. Defining these multi-stage strategies requires making a series of decisions, which may potentially impact inferences, but have not been critically evaluated. We begin by identifying key features of strategies, introducing descriptive terms when they did not already exist in the literature. Strategies differ in how they define and order model building stages. Sequential-by-sub-model strategies focus on one sub-model (parameter) at a time with modeling of subsequent sub-models dependent on the selected sub-model structures from the previous stages. Secondary candidate set strategies model sub-models independently and combine the top set of models from each sub-model for selection in a final stage. Build-up approaches define stages across sub-models and increase in complexity at each stage. Strategies also differ in how the top set of models is selected in each stage and whether they use null or more complex sub-model structures for non-target sub-models. We tested the performance of different model selection strategies using four data sets and three model types. For each data set, we determined the "true" distribution of AIC weights by fitting all plausible models. Then, we calculated the number of models that would have been fitted and the portion of "true" AIC weight we recovered under different model selection strategies. Sequential-by-sub-model strategies often performed poorly. Based on our results, we recommend using a build-up or secondary candidate sets, which were more reliable and carrying all models within 5-10 AIC of the top model forward to subsequent stages. The structure of non-target sub-models was less important. Multi-stage approaches cannot compensate for a lack of critical thought in selecting covariates and building models to represent competing a priori hypotheses. However, even when competing hypotheses for different sub-models are limited, thousands or more models may be possible so strategies to explore candidate model space reliably and efficiently will be necessary.

Remarkable response of native fishes to invasive trout suppression varies with trout density, temperature, and annual hydrology

“…We took a multistage approach to model development and selection whereby competing models representing a priori hypotheses were developed following selection of the best combination of submodels for each variable. This multistage approach was expected to yield the closest result to "true" parsimony as if all combinations of plausible models were fitted and compared (Morin et al 2020). In the first stage, we compared up to six models for each variable with the intercept-only model, with (i) the single predictor included in the count side of the model and an intercept only in the binomial model, and random intercepts; (ii) the predictor included only on the binomial model, and random intercepts; (iii) the predictor on both count and binomial elements of the model and random intercepts; and (iv-vi) repeating the above models with the exception that the models included random slope interactions with trout density metrics.…”

“…Bayesian information criteria (BIC) scores were used to compare models (BICtab function, R package bbmle; Bolker and R Core Team 2017), which we expected would select for models with the strongest relationship with native fish distribution and abundance (Burnham and Anderson 2002;Aho et al 2014). All single-variable models within ⌬5 BIC of the top model were carried forward into the next model selection stage (Morin et al 2020).…”

Multi‐year nest box occupancy and short‐term resilience to wildfire disturbance by barn owls in a vineyard agroecosystem

“…We built our candidate model set using a secondary candidate set strategy where we created a candidate model set for each sub‐model (initial occupancy, colonization, and extinction) before using the top models from each sub‐model set for a final candidate model set (Morin et al 2020). Each candidate sub‐model represented a different hypothesis about which combination of land cover, nest box design, and wildfire predictors would best explain each parameter in the multi‐season occupancy model.…”

Responses to land cover and grassland management vary across life‐history stages for a grassland specialist

“…We modeled home range, brood‐rearing, and nest‐site selection using a multistage model selection approach that incorporates elements of the build‐up and secondary subsets approaches recommended by Morin et al. (2020). For each of the three life‐history stages (hen, nest, and brood), we first examined land cover and management variables in separate subsets and ranked models based on Akaike's information criterion corrected for small sample size (AIC c ; Burnham & Anderson, 2002).…”