What Works Best and When: Accounting for Multiple Sources of Pureselection Bias in Program Evaluations

Jung, Haeil; Pirog, Maureen A.

doi:10.1002/pam.21764

Cited by 7 publications

(4 citation statements)

References 41 publications

(90 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More than 70 WSCs have been conducted to date, including at least three in this journal (Bifulco, ; Wilde & Hollister, ; Wing & Cook, ). Two other studies in this journal used somewhat comparable methods (Jung & Pirog, , ). Most WSCs test how effective design and analysis methods are for reducing the population differences that cloud the causal interpretation of nonexperimental data.…”

Section: Introductionmentioning

confidence: 99%

The Internal and External Validity of the Regression Discontinuity Design: A Meta‐analysis of 15 Within‐study Comparisons

Chaplin¹,

Cook²,

Zurovac³

et al. 2018

J Policy Anal Manage

View full text Add to dashboard Cite

Theory predicts that regression discontinuity (RD) provides valid causal inference at the cutoff score that determines treatment assignment. One purpose of this paper is to test RD's internal validity across 15 studies. Each of them assesses the correspondence between causal estimates from an RD study and a randomized control trial (RCT) when the estimates are made at the same cutoff point where they should not differ asymptotically. However, statistical error, imperfect design implementation, and a plethora of different possible analysis options, mean that they might nonetheless differ. We test whether they do, assuming that the bias potential is greater with RDs than RCTs. A second purpose of this paper is to investigate the external validity of RD by exploring how the size of the bias estimates varies across the 15 studies, for they differ in their settings, interventions, analyses, and implementation details. Both Bayesian and frequentist meta‐analysis methods show that the RD bias is below 0.01 standard deviations on average, indicating RD's high internal validity. When the study‐specific estimates are shrunken to capitalize on the information the other studies provide, all the RD causal estimates fall within 0.07 standard deviations of their RCT counterparts, now indicating high external validity. With unshrunken estimates, the mean RD bias is still essentially zero, but the distribution of RD bias estimates is less tight, especially with smaller samples and when parametric RD analyses are used.

show abstract

Section: Introductionmentioning

confidence: 99%

The Internal and External Validity of the Regression Discontinuity Design: A Meta‐analysis of 15 Within‐study Comparisons

Chaplin¹,

Cook²,

Zurovac³

et al. 2018

J Policy Anal Manage

View full text Add to dashboard Cite

show abstract

“…While we cannot mitigate or even test for self‐selection bias directly, we conduct one supplemental analysis to verify our results. Our methodological strategy may also partially pre‐empt this concern, as Jung and Pirog (2014) show that fixed effect estimators reduce bias arising from self‐selection.…”

Section: Methodsmentioning

confidence: 99%

Information sharing and state revenue forecasting performance

Spreen

Guzman

2022

Public Budgeting & Finance

View full text Add to dashboard Cite

This study evaluates whether intergovernmental information sharing enhances forecasting performance. This is accomplished by examining the accuracy of state revenue forecasts following the federal passage of the Tax Cuts and Jobs Act (TCJA) of 2017. The quantitative analysis suggests that states that shared information produced more accurate corporate income tax forecasts than nonsharing states. This result is consistent with surveys and interviews of federal and state officials that reported significant information‐sharing activity arising from uncertainty about the TCJA's corporate income tax provisions. This study demonstrates that information sharing plays an important yet overlooked role in mitigating forecast uncertainty.

show abstract

“…Large governments that select into training likely possess greater financial resources and increased administrative capacity compared with smaller governments that did not participate. For this reason, it is likely that the analytical results would be confounded by self-selection bias absent the assumptions inherent in a DD design, although recent methodological work suggests that the inclusion of fixed effects helps to mitigate this bias (Jung & Pirog, 2014). We separately address this concern by enforcing a region of common support containing "treated" governments that participated in training and comparable governments that did not.…”

Section: Methodsmentioning

confidence: 99%

Can Employee Training Influence Local Fiscal Outcomes?

Spreen

Afonso

Gerrish

2020

The American Review of Public Administration

View full text Add to dashboard Cite

Employee training is often viewed as essential for incorporating performance management practices into public organizations, but few studies directly link training programs to subsequent changes in organizational outcomes. Typically, evaluations of the impact of training and management innovations more broadly focuses narrowly on improvements at the mean of the distribution, ignoring isomorphic pressures that may spur divergent responses at opposite tails of the distribution. We examine these notions by testing whether training local government personnel on the use of financial performance information in decision-making influences fiscal outcomes. Specifically, we compare the outcomes of North Carolina local governments whose employees participated in training on a new fiscal benchmarking tool at the University of North Carolina School of Government to peer governments that did not participate. Municipal governments with at least one trained employee experienced modest changes, on average, across most of the financial ratios reported in the benchmarking tool. By comparison, the dispersion of the reported outcomes declined considerably among municipal governments whose employees participated in training in comparison to control governments. The strength of this response increased with the number of public officials trained. The results indicate that employee training can facilitate the use of performance benchmarking systems in public sector decision-making. They also suggest that benchmarking without explicit performance targets may encourage convergence toward the average outcome.

show abstract

What Works Best and When: Accounting for Multiple Sources of Pureselection Bias in Program Evaluations

Cited by 7 publications

References 41 publications

The Internal and External Validity of the Regression Discontinuity Design: A Meta‐analysis of 15 Within‐study Comparisons

The Internal and External Validity of the Regression Discontinuity Design: A Meta‐analysis of 15 Within‐study Comparisons

Information sharing and state revenue forecasting performance

Can Employee Training Influence Local Fiscal Outcomes?

Contact Info

Product

Resources

About