Towards Query Performance Prediction for Neural Information Retrieval: Challenges and Opportunities

Faggioli, Guglielmo; Formal, Thibault; Lupart, Simon; Marchesin, Stefano; Clinchant, Stephane; Ferro, Nicola; Piwowarski, Benjamin

doi:10.1145/3578337.3605142

Cited by 6 publications

(2 citation statements)

References 92 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These studies notably include [55,56]. Recently, [57,58] have raised the question of the effectiveness of evaluating the difficulty of Neural Information Retrieval based on PLM (Pre-trained Language Models).…”

Section: Difficulty In Information Retrievalmentioning

confidence: 99%

Sentiment Difficulty in Aspect-Based Sentiment Analysis

Chifu,

Fournier

2023

Mathematics

View full text Add to dashboard Cite

Subjectivity is a key aspect of natural language understanding, especially in the context of user-generated text and conversational systems based on large language models. Natural language sentences often contain subjective elements, such as opinions and emotions, that make them more nuanced and complex. The level of detail at which the study of the text is performed determines the possible applications of sentiment analysis. The analysis can be done at the document or paragraph level, or, even more granularly, at the aspect level. Many researchers have studied this topic extensively. The field of aspect-based sentiment analysis has numerous data sets and models. In this work, we initiate the discussion around the definition of sentence difficulty in this context of aspect-based sentiment analysis. To assess and quantify the difficulty of the aspect-based sentiment analysis, we conduct an experiment using three data sets: “Laptops”, “Restaurants”, and “MTSC” (Multi-Target-dependent Sentiment Classification), along with 21 learning models from scikit-learn. We also use two textual representations, TF-IDF (Terms frequency-inverse document frequency) and BERT (Bidirectional Encoder Representations from Transformers), to analyze the difficulty faced by these models in performing aspect-based sentiment analysis. Additionally, we compare the models with a fine-tuned version of BERT on the three data sets. We identify the most challenging sentences using a combination of classifiers in order to better understand them. We propose two strategies for defining sentence difficulty. The first strategy is binary and considers sentences as difficult when the classifiers are unable to correctly assign the sentiment polarity. The second strategy uses a six-level difficulty scale based on how many of the top five best-performing classifiers can correctly identify sentiment polarity. These sentences with assigned difficulty classes are then used to create predictive models for early difficulty detection. The purpose of estimating the difficulty of aspect-based sentiment analysis is to enhance performance while minimizing resource usage.

show abstract

Section: Difficulty In Information Retrievalmentioning

confidence: 99%

Sentiment Difficulty in Aspect-Based Sentiment Analysis

Chifu,

Fournier

2023

Mathematics

View full text Add to dashboard Cite

show abstract

“…Bridging this gap, our proposed web-based system, DenseQuest, employs a diverse range of unsupervised performance evaluation methods (which we refer to as DR selection methods). These methods can be categorized as either per-query (query performance prediction, or QPP [4]) or per-collection [5]. QPP approaches aim to predict the performance of each individual query within a model, whereas per-collection methods seek to predict the average performance across all queries, focusing on model comparison rather than query comparison.…”

Section: Introductionmentioning

confidence: 99%

Embark on DenseQuest: A System for Selecting the Best Dense Retriever for a Custom Collection

Khramtsova,

Leelanupab,

Zhuang

et al. 2024

Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

In this demo we present a web-based application for selecting an effective pre-trained dense retriever to use on a private collection. Our system, DenseQuest, provides unsupervised selection and ranking capabilities to predict the best dense retriever among a pool of available dense retrievers, tailored to an uploaded target collection. DenseQuest implements a number of existing approaches, including a recent, highly effective method powered by Large Language Models (LLMs), which requires neither queries nor relevance judgments. The system is designed to be intuitive and easy to use for those information retrieval engineers and researchers who need to identify a general-purpose dense retrieval model to encode or search a new private target collection. Our demonstration illustrates conceptual architecture and the different use case scenarios of the system implemented on the cloud, enabling universal access and use. DenseQuest is available at https://densequest.ielab.io. CCS CONCEPTS• Information systems → Evaluation of retrieval results.

show abstract