Focal elements of neural information retrieval models. An outlook through a reproducibility study

Marchesin, Stefano; Purpura, Alberto; Silvello, Gianmaria

doi:10.1016/j.ipm.2019.102109

Cited by 19 publications

(8 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We focus on systems based on WEs, leaving aside lexical ones, since our interest is to evaluate the impact of debiasing on classical performance measures. Our results, which are in line with prior art [49], show that debiasing (both regular and strong) produce negligible changes to average performance.…”

Section: Debiasing Moderatly Reduces Gsrsupporting

confidence: 89%

Gender Stereotype Reinforcement: Measuring the Gender Bias Conveyed by Ranking Algorithms

Fabris¹,

Purpura²,

Silvello³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Search Engines (SE) have been shown to perpetuate well-known gender stereotypes identified in psychology literature and to influence users accordingly. Similar biases were found encoded in Word Embeddings (WEs) learned from large online corpora. In this context, we propose the Gender Stereotype Reinforcement (GSR) measure, which quantifies the tendency of a SE to support gender stereotypes, leveraging gender-related information encoded in WEs.Through the critical lens of construct validity, we validate the proposed measure on synthetic and real collections. Subsequently, we use GSR to compare widely-used Information Retrieval ranking algorithms, including lexical, semantic, and neural models. We check if and how ranking algorithms based on WEs inherit the biases of the underlying embeddings. We also consider the most common debiasing approaches for WEs proposed in the literature and test their impact in terms of GSR and common performance measures. To the best of our knowledge, GSR is the first specifically tailored measure for IR, capable of quantifying representational harms.

show abstract

Section: Debiasing Moderatly Reduces Gsrsupporting

confidence: 89%

Gender Stereotype Reinforcement: Measuring the Gender Bias Conveyed by Ranking Algorithms

Fabris¹,

Purpura²,

Silvello³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Using reproducible -and thus trustworthy -statistical tools is crucial to drawing robust inferences and conclusions. In recent years, many fields have devoted a lot of effort to reproducing and generalizing their systems and algorithms [60,106,56,36]. Yet, the literature still lacks reproducibility studies on the statistical tools used to compare the performance of such systems and algorithms.…”

Section: Introductionmentioning

confidence: 99%

Modelling and Explaining IR System Performance Towards Predictive Evaluation

Faggioli

2023

SIGIR Forum

View full text Add to dashboard Cite

Information Retrieval (IR) systems play a fundamental role in many modern commodities, including search engines (SEs), digital libraries, recommender systems, and social networks. The IR task is particularly challenging because of the volatility of IR systems performance: users' information needs change daily, and so do the documents to be retrieved and the concept of what is relevant to a given information need. Therefore, the empirical offline evaluation of an IR system is a costly and slow post-hoc procedure, that happens after the system deployment. Given the challenges linked to empirical IR evaluation, predicting a system's performance before its deployment would add significant value to the development of an IR system. In this manuscript, we place the cornerstone for the prediction of IR performance, by considering two closely related areas: the modeling of IR systems performance and the Query Performance Prediction (QPP). The former area allows us to identify those features that impact the most on the performance and that can be used as predictors, while the latter provides us with a starting point to instantiate the predictive task in IR. Concerning the modeling of IR performance, we first investigate one of the most popular statistical tools, ANOVA. In particular, we compare traditional ANOVA with a recent approach, bootstrap ANOVA, and observe the different conclusions that can be achieved using these two different statistical tools [Faggioli and Ferro, 2021]. Secondly, using ANOVA, we study the concept of topic difficulty and observe that the topic difficulty is not an intrinsic property of the information need but stems from the formulation used to represent the topic [Culpepper et al., 2022]. Finally, we show how to use Generalized Linear Models (GLMs) as an alternative to the traditional linear modeling of IR performance [Faggioli et al., 2022]. We show how GLMs provide more powerful inference with comparable stability. Our analyses on the QPP domain start with developing a predictor used to select among a set of reformulations for the same information need, the best-performing one for the systematic review task [Di Nunzio and Faggioli, 2021]. Secondly, we investigate how to classify queries as either semantic or lexical to predict whether neural models will perform better than lexical ones. Finally, given the challenges shown in the evaluation of the previous approaches, we devise a new evaluation procedure, dubbed sMARE [Faggioli et al., 2021]. sMARE allows moving from a single point estimation of the performance to a distributional one, allowing us to achieve improved comparisons between QPP models and more precise analyses. Awarded by: University of Padova, Padova, Italy on 20 March 2023. Supervised by: Nicola Ferro. Available at: https://www.research.unipd.it/handle/11577/3472979.

show abstract

“…The feature selection problem is mitigated by deep learning and, more generally, neural approaches that have gained popularity in recent years. Despite these methods being extremely versatile and generally able to provide good overall effectiveness, it is known their performance is not always stable and may vary a lot across topics, for example the performance may improve for half of the topics while degrade for the other half [30]. A further disadvantage is that these neural approaches are very demanding in terms of computing resources and require enormous amounts of data which leads to larger and larger models that are not free from risks, as pointed out by Bender et al [6].…”

Section: Introductionmentioning

confidence: 99%

Towards Feature Selection for Ranking and Classification Exploiting Quantum Annealers

Dacrema

Moroni

Nembrini

et al. 2022

Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Feature selection is a common step in many ranking, classification, or prediction tasks and serves many purposes. By removing redundant or noisy features, the accuracy of ranking or classification can be improved and the computational cost of the subsequent learning steps can be reduced. However, feature selection can be itself a computationally expensive process. While for decades confined to theoretical algorithmic papers, quantum computing is now becoming a viable tool to tackle realistic problems, in particular special-purpose solvers based on the Quantum Annealing paradigm. This paper aims to explore the feasibility of using currently available quantum computing architectures to solve some quadratic feature selection algorithms for both ranking and classification.The experimental analysis includes 15 state-of-the-art datasets. The effectiveness obtained with quantum computing hardware is comparable to that of classical solvers, indicating that quantum computers are now reliable enough to tackle interesting problems. In terms of scalability, current generation quantum computers are able to provide a limited speedup over certain classical algorithms and hybrid quantum-classical strategies show lower computational cost for problems of more than a thousand features. CCS CONCEPTS• Information systems → Content analysis and feature selection; • Computer systems organization → Quantum computing.

show abstract

Focal elements of neural information retrieval models. An outlook through a reproducibility study

Cited by 19 publications

References 17 publications

Gender Stereotype Reinforcement: Measuring the Gender Bias Conveyed by Ranking Algorithms

Gender Stereotype Reinforcement: Measuring the Gender Bias Conveyed by Ranking Algorithms

Modelling and Explaining IR System Performance Towards Predictive Evaluation

Towards Feature Selection for Ranking and Classification Exploiting Quantum Annealers

Contact Info

Product

Resources

About