Crowdsourcing Accurately and Robustly Predicts Supreme Court Decisions

Abstract

Scholars have increasingly investigated "crowdsourcing" as an alternative to expert-based judgment or purely data-driven approaches to predicting the future. Under certain conditions, scholars have found that crowdsourcing can outperform these other approaches. However, despite interest in the topic and a series of successful use cases, relatively few studies have applied empirical model thinking to evaluate the accuracy and robustness of crowdsourcing in real-world contexts. In this paper, we offer three novel contributions. First, we explore a dataset of over 600,000 predictions from over 7,000 participants in a multi-year tournament to predict the decisions of the Supreme Court of the United States. Second, we develop a comprehensive crowd construction framework that allows for the formal description and application of crowdsourcing to real-world data. Third, we apply this framework to our data to construct more than 275,000 crowd models. We find that in out-of-sample historical simulations, crowdsourcing robustly outperforms the commonly-accepted null model, yielding the highest-known performance for this context at 80.8% case level accuracy. To our knowledge, this dataset and analysis represent one of the largest explorations of recurring human prediction to date, and our results provide additional empirical support for the use of crowdsourcing as a prediction method.

Asking Versus Telling: The Supreme Court’s Strategic Use of Questions and Statements During Oral Arguments

“…Without walking through each of the 16 predictions, this forecasting comparison supports my findings in Table 1 and Figure 2: the Court’s communication behavior provides insight to a party’s chances of winning. It is indeed not a coincidence that Linda Greenhouse, a human expert basing her predictions (at least in part) on oral arguments, soundly beat the panel of human experts who simply examined legal issues, while my simple oral argument variables held their own against the Washington University model and also the newer and more sophisticated models of Dietrich, Enos, and Sen (2019), Katz, Bommarito, and Blackman (2014), and Katz, Bommarito, and Blackman (2017). Quite simply, oral arguments provide valuable data in predicting outcomes.…”

“…In this section, I compare my oral argument, communication-based measure against other forecasts—specifically Marshall+ that uses machine learning and over 90 variables (Katz, Bommarito and Blackman 2014); fantasySCOTUS, a crowd-sourced forecasting Web site (Katz, Bommarito and Blackman 2017); a model containing vocal pitch (Dietrich, Enos and Sen 2019); and The Supreme Court Forecasting Project, which pitted legal experts against a computer model and veteran New York Times Supreme Court correspondent Linda Greenhouse (Washington University Law School 2016). The prediction methods vary widely in complexity and source—some relying purely on human expertise, some purely on modeling, and others employing a marriage of the two—and so provide a robust test of my measure’s usefulness.…”

“…7–8). This type of crowd-sourced modeling, however, only “beats” my simple logistic regression when Katz, Bommarito, and Blackman (2017) employ experience thresholding along with a follow-the-leader model—a model that weighs all guesses equally predicts 66.4% and 65.5% of outcomes and votes, highlighting the importance of human expertise. To wit, while oral argument behavior wins out over averaged crowdsourcing, crowdsourcing prevails when human expertise is weighted.…”

Artificial Intelligence and International Arbitration

“…To begin, the study's objective was to develop a model that would be generally and consistently relevant to all US Supreme Court judgments across time, not simply in a particular year or with a particular composition of the Court. 43 Second, the research followed the idea that 'all information required for the model to produce an estimate should be knowable prior to the date of the decision'. As explained above, this is to guarantee that the model is capable of ex ante result prediction.…”

An Artificial-Intelligence-Based Semantic Assist Framework for Judicial Trials

“…The machine-learning method extracts the text-information structure according to the legal elements, labels them according to a knowledge graph, and constructs the supervised-learning-based model. 9 The interpretation of this category of method did not conform to the legal logic of the judicial-trial process and lacks judicial interpretability.…”