Quality Review of Mass Adjudication: A Randomized Natural Experiment at the Board of Veterans Appeals, 2003–16

Abstract

We study a unique natural experiment, during which 5-10% of draft opinions by judges of the Board of Veterans Appeals (BVA) were randomly selected for "quality review" by a team of full-time staff attorneys for nearly 15 years. This performance program had the express goals of measuring accuracy and reducing reversal rates on appeal. In cases of legal error, the quality review team wrote memoranda to judges to permit correction before opinions were issued. We use rich internal administrative data on nearly 600,000 cases from 2002-2016 to provide the first rigorous study of this review process. With precise estimates, we show that the program had no appreciable effect on reducing appeals or reversals. Based on internal records, we demonstrate that this inefficacy is likely by design, as meeting the performance measure of "accuracy" was at cross-purposes with error correction. These findings inform longstanding questions of law, organization, and bureaucracy, including performance management, standards of review, and the institutional design of mass adjudication.

Context-aware legal citation recommendation using deep learning

“…Due to these case volumes and constrained resources, the BVA suffers from both a large backlog of cases and large error rates in decisions. Roughly 15% of (single-issue) cases are appealed and around 72% of appealed cases are reversed or remanded by a higher court [14]. These challenges are typical for agencies like the Social Security Administration, the Office of Medicare Hearings and Appeals, and the immigration courts, which adjudicate far more cases than all federal courts combined.…”