VoMBaT: A Tool for Visualising Evaluation Measure Behaviour in High-Recall Search Tasks

Abstract

The objective of High-Recall Information Retrieval (HRIR) is to retrieve as many relevant documents as possible for a given search topic. One approach to HRIR is Technology-Assisted Review (TAR), which uses information retrieval and machine learning techniques to aid the review of large document collections. TAR systems are commonly used in legal eDiscovery and systematic literature reviews. Successful TAR systems are able to find the majority of relevant documents using the least number of assessments. Commonly used retrospective evaluation assumes that the system achieves a specific, fixed recall level first, and then measures the precision or work saved (e.g., precision at r% recall). This approach can cause problems related to understanding the behaviour of evaluation measures in a fixed recall setting. It is also problematic when estimating time and money savings during technology-assisted reviews.This paper presents a new visual analytics tool to explore the dynamics of evaluation measures depending on recall level. We implemented 18 evaluation measures based on the confusion matrix terms, both from general IR tasks and specific to TAR. The tool allows for a comparison of the behaviour of these measures in a fixed recall evaluation setting. It can also simulate savings in time and money and a count of manual vs automatic assessments for different datasets depending on the model quality. The tool is open-source, and the demo is available under the following URL: https://vombat.streamlit.app. CCS CONCEPTS• Information systems → Evaluation of retrieval results; Retrieval effectiveness; • Human-centered computing → Visual analytics.

An analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews

“…When the screening problem is treated as a ranking task, such as screening prioritisation or stopping prediction; the performance is measured in terms of rank-based metrics and metrics at a fixed cut-off, such as nDCG@n, P recision@n, and last relevant found [69,28]. On the other hand, when the screening problem is treated as a classification task, the performance in this case is measured based on the confusion matrix and the notions of Precision and Recall are commonly used [41,59]. One challenge arising from these two distinct approaches is the difficulty in going beyond simple effectiveness measures and comparing the real-world savings for users.…”

“…We evaluate models using nDCG@10, M AP , Recall at rank k with k in {10, 50, 100} (R@k). Additionally, we compute three measures specifically designed for the task of CS: True Negative Rate at 95% Recall (T N R@95%) [40,41], normalised Precision at 95% Recall (nP @95%) [41], and average position at which the last relevant item is found [30,31,32], calculated as a percentage of the dataset size, where a lower value indicates better performance (Last Rel).…”

“…The True Negative Rate (T N R) was proposed as an alternative as it addresses some of the limitations of WSS regarding averaging scores from multiple datasets [40]. The measures of normalised Precision at r% recall (nPrecision@r%) and normalised rectified TNR at r% recall (nReTNR@r%) have also been introduced to focus on other important aspects of screening task: screening full texts and estimating users' time savings when compared to the random ranking, respectively [41].…”

CRUISE-Screening: Living Literature Reviews Toolbox

“…Automated citation screening is an umbrella term for using NLP, machine learning and information retrieval (IR) techniques with the goal of decreasing the time spent on manual screening. Classification approaches train a supervised model on an annotated dataset to determine whether a paper should be included or excluded from the review [23,24].…”