Constrained statistical inference: sample-size tables for ANOVA and regression

Abstract

Researchers in the social and behavioral sciences often have clear expectations about the order/direction of the parameters in their statistical model. For example, a researcher might expect that regression coefficient β1 is larger than β2 and β3. The corresponding hypothesis is H: β1 > {β2, β3} and this is known as an (order) constrained hypothesis. A major advantage of testing such a hypothesis is that power can be gained and inherently a smaller sample size is needed. This article discusses this gain in sample size reduction, when an increasing number of constraints is included into the hypothesis. The main goal is to present sample-size tables for constrained hypotheses. A sample-size table contains the necessary sample-size at a pre-specified power (say, 0.80) for an increasing number of constraints. To obtain sample-size tables, two Monte Carlo simulations were performed, one for ANOVA and one for multiple regression. Three results are salient. First, in an ANOVA the needed sample-size decreases with 30–50% when complete ordering of the parameters is taken into account. Second, small deviations from the imposed order have only a minor impact on the power. Third, at the maximum number of constraints, the linear regression results are comparable with the ANOVA results. However, in the case of fewer constraints, ordering the parameters (e.g., β1 > β2) results in a higher power than assigning a positive or a negative sign to the parameters (e.g., β1 > 0).

Small Sample Size Solutions

“…Hypotheses with order constraints ('<' and '>') are also referred to as order constrained hypotheses. Such informative hypotheses can be compared to each other by means of an F-bar test (Silvapulle & Sen, 2004;Vanbrabant & Rosseel, 2020;Vanbrabant, Van de Schoot, & Rosseel, 2015) or with Bayes factors , which are used for the method in this chapter. Bayes factors are defined in Bayes' theorem, which describes how knowledge about the relative belief in hypotheses can be updated with evidence in data:…”

“…However, contrast tests are not the same as informative hypothesis tests (Baayen, Klugkist, & Mechsner, 2012). Third, incorporating order constraints in the analysis will result in substantially more power (e.g., Bartholomew, 1961aBartholomew, , 1961bKuiper & Hoijtink, 2010;Perlman, 1969;Robertson, Wright, & Dykstra, 1988;Vanbrabant, Van de Schoot, & Rosseel, 2015;Van de Schoot & Strohmeier, 2011). Vanbrabant et al (2015) showed that using ordered means and multiple one-sided regression coefficients yields adequate power with 50% of the sample size required by ANOVA and regression (respectively).…”

How do included and excluded students with SEBD function socially and academically after 1,5 year of special education services?

“…students with SEBD have often been limited by small sample sizes (e.g., Lane et al, 2005), Bayesian statistics have provided possibilities to handle small samples with greater accuracy. That is, testing specific constrained hypotheses against each other will lead to increased power and will decrease the need for large sample sizes (see for a more elaborate explanation Vanbrabant, Van de Schoot, & Rosseel, 2015). Specifically, instead of testing all possible solutions in a parameter space, we tested only a predetermined set of solutions in the parameter spacenamely specific constrained hypotheses based on previous literaturewhich is easier to falsify or to find support for than when you do not have specified any hypotheses at all.…”

The Pervasive Avoidance of Prospective Statistical Power: Major Consequences and Practical Solutions

“…Vanbrabant et al (2015), offer sample-size tables for ANOVA and regression when using Constrained statistical inference.…”