Problems with Probability

Abstract

Some countries have explored the idea of using artificial intelligence (AI) systems to help triage the backlog of cases and facilitate the resolution of civil disputes. In theory, AI can accomplish this by establishing the facts of cases and predicting the outcomes of disputes. But the use of AI in the courtroom gives rise to new problems. AI technologies help solve prediction problems. These solutions are typically expressed as probabilities. How should judges incorporate these predictions in their decision making? There is no obviously correct approach for converting probabilistic predictions of legal outcomes into binary legal decisions. Any approach that does so has benefits and drawbacks. Importantly, a balance of probabilities approach – where liability is established if the AI predicts a likelihood of liability greater than 50 per cent and not otherwise – is not suitable when converting a predicted outcome into an actual outcome. Adopting this approach would significantly alter the outcomes of legal cases and have a dramatic and disruptive effect upon the law. The most notable disruption would be observed in settlement behaviour and outcomes.

AI and the nature of disagreement

“…Using such algorithmic decisions to replicate previous decisions would likely yield more consistent results than the decisions of human judges. Some countries have already explored implementing AI systems to help triage caseload of civil disputes (see [14]).…”

“…Using algorithms based on previous decisions, if used categorically, could calcify the common law, potentially limiting their usefulness in new or rapidly evolving legal areas [48]. And difficult problems emerge when turning probabilistic predictions into binary legal outcomes [13,14].…”

“…And the manner of the conversion may depend on the type of disagreement. That is, there may be good reasons to convert probabilities over facts into binary findings differently to the way we convert probabilities over outcomes into binary rulings (see [14]).…”