Rank tests in unmatched clustered randomized trials applied to a study of teacher training

Abstract

In the Teacher and Leader Performance Evaluation Systems study, schools were randomly assigned to receive new measures of teacher and principal performance. One outcome in the study, measured at the teacher level, was truncated at zero, and displayed a long tail. Rank-based statistics are one natural method to apply to such outcomes, since inferences will be robust and exact, and we can avoid assumptions about the model that generated the data. We investigate four different possible rank statistics that vary in the form of weighting applied to clusters. Each test statistic has the correct level but may vary in terms of the power to detect departures from the null. We conduct a simulation for power comparing to linear mixed models with Normal, t, and Cauchy errors. We obtain a point estimate and construct confidence intervals by applying the Tobit model of effects, which assumes that treatment increases the outcome by a constant amount but only if the response under control would be positive. We also develop a formal randomization-based method for testing the appropriateness of the Tobit model of effects. In the data from the study, we find no evidence against the Tobit model of effects.

Model-Assisted Analyses of Cluster-Randomized Experiments

“…This approach requires strong modelling assumptions, and it is a curious research topic to study its design‐based properties. Another alternative approach is to use the Fisher randomization tests that yield finite‐sample exact p ‐values (Ding & Keele, 2018; Gail et al., 1996; Zhang et al., 2012). This approach often restricts the model of the individual treatment effect, for example, testing the sharp null hypothesis.…”