Sensitivity and Specificity of Information Criteria

Dziak, John J.; Coffman, Donna L.; Lanza, Stephanie T.; Li, Runze; Jermiin, Lars S.

doi:10.1101/449751

Cited by 143 publications

(148 citation statements)

References 127 publications

Supporting

Mentioning

144

Contrasting

Unclassified

Order By: Relevance

“…the average marginal effect), those effects were negligible in comparison with completely spurious effects arising from random allocation of species to islands (Tables S4.6–S4.8 in Appendix S4). As discussed in a recent study by Dziak, Coffman, Lanza, Li, and Jermiin () there is also a tendency of AIC to prioritize the avoidance of “false negatives” with respect to variable selection, and we deem it equally plausible that the apparent contradiction can be attributed to AIC model “overfitting”.…”

Section: Discussionmentioning

confidence: 75%

Bryophyte community assembly on young land uplift islands – Dispersal and habitat filtering assessed using species traits

Tiselius

Lundbäck

Lönnell

et al. 2019

Journal of Biogeography

View full text Add to dashboard Cite

Aim To assess habitat filtering and dispersal limitation in spore plant community assembly using bryophytes on recently emerged land uplift islands as study system. Location Gulf of Bothnia, northern Europe. Taxa Bryophytes, including the spore plant phyla Bryophyta (mosses) and Marchantiophyta (liverworts). Methods The species compositions of 20 coastal land uplift islands differing in age, area, connectivity and habitat composition were recorded in the field. In addition, we compiled a list of the regional species pool (446 species) and gathered data on species traits related to habitat affiliations (substrate, light, moisture, and pH) and dispersal capacity (regional abundance, spore size, sporophyte frequency, sexual system, vegetative propagules). For the 420 species with available trait data, we used multivariate generalized linear models to compare trait effects on species occurrence probabilities on the islands. Results Occurrence probabilities depended strongly on habitat affiliations. In addition, occurrence probabilities were lower for predominantly asexual species than for sexual species and for regionally rare than for regionally abundant species. Having specialized asexual propagules increased occurrence probabilities, but compensated only partly for the reductions in asexual species. No effect of the size of sexually produced spores was detected. Comparison of trait effects across island size and connectivity gradients revealed (a) reduced habitat filtering on larger islands and (b) decreasing negative effects of being predominantly asexual with increasing island connectivity. Conclusions Both habitat filtering and dispersal capacities affect the community assembly of spore plants on land uplift islands. Asexual mosses and liverworts show landscape scale (≤10 km) dispersal limitation. The weak or absent relationships between island connectivity and the effects of dispersal traits suggest that colonization is regulated mainly by habitat availability and the abundance of each species in a “regional spore rain” from which colonists are recruited.

show abstract

Section: Discussionmentioning

confidence: 75%

Bryophyte community assembly on young land uplift islands – Dispersal and habitat filtering assessed using species traits

Tiselius

Lundbäck

Lönnell

et al. 2019

Journal of Biogeography

View full text Add to dashboard Cite

show abstract

“…All models were reduced using a bidirectional stepwise selection of confounders determined by the Akaike information criterion in an attempt to identify a model best balancing accuracy and simplicity, while maintaining a low false negative rate . The vital sign deviation under analysis was forced to remain in the list of final confounders.…”

Section: Methodsmentioning

confidence: 99%

Integrating intraoperative physiology data into outcome analysis for the ACS Pediatric National Surgical Quality Improvement Program

Görges

Afshar

West

et al. 2018

Pediatric Anesthesia

View full text Add to dashboard Cite

Summary Background The Pediatric National Surgical Quality Improvement Program (P‐NSQIP) samples surgical procedures for benchmarking and quality improvement. While generally comprehensive, P‐NSQIP does not collect intraoperative physiologic data, despite potential impact on outcomes. Aims The aims of this study were (a) to describe a methodology to augment P‐NSQIP with vital signs data and (b) demonstrate its utility by exploring relationships that intraoperative hypothermia and hypotension have with P‐NSQIP outcomes. Methods Vital signs from 2012 to 2016 were available in a research databank. Episodes of hypotension and hypothermia were extracted and recorded alongside local P‐NSQIP data. Multivariable regression analyses were performed to explore associations with undesired outcomes, including: surgical site infection, wound disruption, unplanned return to the operating room, and blood transfusion. Model variables were selected with the Akaike information criterion using 2012‐2014 as the training set and validated with receiver operating characteristics analysis using 2015‐2016 as the testing set. Results Data from 6737 patients were analyzed, with 43.9% female, median [interquartile range] age 5.8 [1.3‐12.4] years, undergoing procedures lasting 118 [75‐193] minutes. Hypothermia, observed in 45% of cases, was associated with wound disruption (odds ratio 1.75, 95% CI 1.1‐2.83). Hypotension, observed in 60% of cases, was associated with unplanned returns (odds ratio 1.58, 95% CI 1.02‐2.51), and transfusions (odds ratio 1.95, 95% CI 1.14‐3.52). Surgical site infection, wound disruption, unplanned return, and transfusion models had areas under the receiver operating characteristic curve of 0.69/0.67, 0.59/0.63, 0.78/0.79, and 0.92/0.93 for validation models including hypothermia/hypotension respectively. Conclusion Adding intraoperative vital signs to P‐NSQIP data allowed identification of two modifiable risk factors: hypothermia was associated with increased wound disruption, and hypotension with increased blood transfusions and unplanned returns to the operating room. These findings may motivate prospective studies and prompt other centers and P‐NSQIP to augment outcome data with intraoperative physiological data.

show abstract

“…Attention to type I error is purportedly emphasized over attention to type II error in classic hypothesis testing – although BIC generally has far lower rates of type I error than FSTs – because the former constitutes an incorrect statement, while the latter is merely a ‘failure to reject’ (Dziak et al . ). De‐emphasis of type II error, however, may result in underfitting and loss of predictive power.…”

Section: Discussionmentioning

confidence: 97%

“…The ordering of demarcation lines is in agreement with Dziak et al . () who previously defined AIC and BIC as methods that emphasize sensitivity (statistical power) and specificity (avoidance of type I error) in null hypothesis tests, respectively.…”

Section: A Graphical Heuristicmentioning

confidence: 99%

A graphical framework for model selection criteria and significance tests: refutation, confirmation and ecology

2016

View full text Add to dashboard Cite

Summary In this study, we use a novel graphical heuristic to compare the way four methods: significance testing, two popular information‐theoretic approaches (AIC and BIC) and Good's Bayes/non‐Bayes compromise (an underutilized hypothesis testing approach whose demarcation criterion adjusts for n), evaluate the merit of competing hypotheses, for example H0 and HA. A primary goal of our work is to clarify the concept of strong consistency in model selection. Explicit considerations of this principle (including the strong consistency of BIC) are currently limited to technical derivations, inaccessible to most ecologists. We use our graphical framework to demonstrate, in simple terms, the strong consistency of both BIC and Good's compromise. Our framework also locates the evaluated metrics (and ICs in general) along a conceptual continuum of hypothesis refutation/confirmation that considers n, parameter number and effect size. Along this continuum, significance testing and particularly AIC are refutative for H0, whereas Good's compromise and particularly BIC are confirmatory for the true hypothesis. Our work graphically demonstrates the well‐known asymptotic bias of significance tests for HA, and the incorrectness of using statistically non‐consistent methods for point hypothesis testing. To address these issues, we recommend: (i) dedicated confirmatory methods with strong consistency like BIC for use in point hypothesis testing and confirmatory model selection; (ii) significance tests for use in exploratory/refutative hypothesis testing, particularly when conjoined with rational approaches (e.g. Good's compromise, power analyses) to account for the effect of n on P‐values; and (iii) asymptotically efficient methods like AIC for exploratory model selection.

show abstract

Sensitivity and Specificity of Information Criteria

Cited by 143 publications

References 127 publications

Bryophyte community assembly on young land uplift islands – Dispersal and habitat filtering assessed using species traits

Bryophyte community assembly on young land uplift islands – Dispersal and habitat filtering assessed using species traits

Integrating intraoperative physiology data into outcome analysis for the ACS Pediatric National Surgical Quality Improvement Program

A graphical framework for model selection criteria and significance tests: refutation, confirmation and ecology

Contact Info

Product

Resources

About