Model Uncertainty and Robustness

Abstract

Model uncertainty is pervasive in social science. A key question is how robust empirical results are to sensible changes in model specification. We present a new approach and applied statistical software for computational multimodel analysis. Our approach proceeds in two steps: First, we estimate the modeling distribution of estimates across all combinations of possible controls as well as specified functional form issues, variable definitions, standard error calculations, and estimation commands. This allows analysts to present their core, preferred estimate in the context of a distribution of plausible estimates. Second, we develop a model influence analysis showing how each model ingredient affects the coefficient of interest. This shows which model assumptions, if any, are critical to obtaining an empirical result. We demonstrate the architecture and interpretation of multimodel analysis using data on the union wage premium, gender dynamics in mortgage lending, and tax flight migration among U.S. states. These illustrate how initial results can be strongly robust to alternative model specifications or remarkably dependent on a knife-edge specification.

Cross-Country Determinants of Early-Stage Necessity and Opportunity-Motivated Entrepreneurship: Accounting for Model Uncertainty

“…This would mean that the investment in these programs may not yield the expected results, and the opportunity costs would also be substantial since scarce funding would then have been diverted from more productive uses. As an attempt to address this concern, this study assesses the robustness of relationships between entrepreneurial action and its antecedent factors, as suggested by prior research, using a methodological approach that has not been applied to this body of research heretofore (Young and Holsteen 2015).…”

“…As Bjørnskov and Foss (2016, p. 301) note, "the entrepreneurship literature has not converged on a consensus on what to consider as a standard or even minimalist empirical specification." This is problematic because theory often does not provide enough guidance to select the proper empirical model (Heckman 2005;Leamer 1983;Raftery 1995;Young and Holsteen 2015). Rather, choosing a model is "difficult, fraught with ethical and methodological dilemmas, and not covered in any serious way in classical statistical texts" (Ho et al 2007, p. 232).…”

“…Recent research has also pointed to a parallel credibility concern in strategic management journals as well (Bergh et al 2017). Such empirical heterogeneity, also known as model uncertainty, presents one of the biggest challenges for applied research (Durlauf et al 2012;Glaeser 2008, Young 2009, Young and Holsteen 2015. It also emphasizes the great need for robustness analyses that can make empirical findings more compelling and less prone to non-robust, trivial, and false-positive results.…”

Revolutionary Leaders and Mass Killing

“…I further examined whether the estimates I have reported are sensitive to adding or deleting other control variables. Using the program developed by Young and Holsteen (2015), I estimated all possible combinations of controls (2,048 models) and store all of the estimates on revolutionary leaders. Figure A1 displays the distribution of all the estimates of revolutionary leaders.…”

Transparency and currency crises

“…To control for potential confounding variables, my primary specifications include several macroeconomic and financial fundamentals that have been suggested as determinants of currency crises: real GDP per capita (logged, from Graham & Tucker, 2017;World Bank, 2015 andFeenstra, Inklaar, &Timmer, 2015), GDP growth (annual % change in GDP per capita, from Graham & Tucker, 2017), trade openness (from Graham & Tucker, 2017), current account balance (from World Bank, 2015), the ratio of foreign reserves to GDP (from Steinberg et al, 2015), the level of exchange rate overvaluation (from Steinberg et al, 2015), the annual rate of inflation (logged, from Steinberg et al, 2015), and the total number of currency crises in other countries in the previous year (from Steinberg et al, 2015). To control for potential confounding variables, my primary specifications include several macroeconomic and financial fundamentals that have been suggested as determinants of currency crises: real GDP per capita (logged, from Graham & Tucker, 2017;World Bank, 2015 andFeenstra, Inklaar, &Timmer, 2015), GDP growth (annual % change in GDP per capita, from Graham & Tucker, 2017), trade openness (from Graham & Tucker, 2017), current account balance (from World Bank, 2015), the ratio of foreign reserves to GDP (from Steinberg et al, 2015), the level of exchange rate overvaluation (from Steinberg et al, 2015), the annual rate of inflation (logged, from Steinberg et al, 2015), and the total number of currency crises in other countries in the previous year (from Steinberg et al, 2015).…”