Evaluating Fairness in Argument Retrieval

Cherumanal, Sachin Pathiyan; Spina, Damiano; Scholer, Falk; Croft, W. Bruce

doi:10.1145/3459637.3482099

Cited by 12 publications

(3 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Viewpoint Diversity in Ranked Outputs. Previous research has shown that search results across topics and domains (e.g., politics [54], health [65,66]) may not always be viewpoint-diverse and that highly-ranked search results are often unbalanced concerning query subtopics [30,50]. Limited diversity, or bias, can root in the overall search result index but become amplified by biased queries and rankings [30,56,66].…”

Section: Related Workmentioning

confidence: 99%

Viewpoint Diversity in Search Results

Draws

Roy

Inel

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

published version features the final layout of the paper including the volume, issue and page numbers. Link to publication General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal.If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the "Taverne" license above, please follow below link for the End User

show abstract

Section: Related Workmentioning

confidence: 99%

Viewpoint Diversity in Search Results

Draws

Roy

Inel

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Participants in the TREC fair ranking track also tested various techniques to produce fair rankings. Many of them utilize diversification-based methods, such as MMR [9], PM-2 and rank fusion [8], and heuristic approaches [14]. Vardasbi et al [1] leveraged LamdaMART [2], ListNET [3], and Logistic regression to maximize evaluation metrics via swapping positions.…”

Section: Related Workmentioning

confidence: 99%

Learn to be Fair without Labels: A Distribution-based Learning Framework for Fair Ranking

Chen,

Fang

2023

Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval

View full text Add to dashboard Cite

Ranking algorithms as an essential component of retrieval systems have been constantly improved in previous studies, especially regarding relevance-based utilities. In recent years, more and more research attempts have been proposed regarding fairness in rankings due to increasing concerns about potential discrimination and the issue of echo chamber. These attempts include traditional score-based methods that allocate exposure resources to different groups using pre-defined scoring functions or selection strategies and learning-based methods that learn the scoring functions based on data samples. Learning-based models are more flexible and achieve better performance than traditional methods. However, most of the learning-based models were trained and tested on outdated datasets where fairness labels are barely available. Stateof-art models utilize relevance-based utility scores as a substitute for the fairness labels to train their fairness-aware loss, where plugging in the substitution does not guarantee the minimum loss. This inconsistency challenges the model's accuracy and performance, especially when learning is achieved by gradient descent. Hence, we propose a distribution-based fair learning framework (DLF) that does not require labels by replacing the unavailable fairness labels with target fairness exposure distributions. Experimental studies on TREC fair ranking track dataset confirm that our proposed framework achieves better fairness performance while maintaining better control over the fairness-relevance trade-off than state-of-art fair ranking frameworks.

show abstract

“…With respect to the evaluation of argument annotations, we could integrate into ARGAEL diverse evaluation methodologies and procedures [40], and metrics, such as the fairness and diversity of arguments [41], which allow measuring the quality of identified arguments beyond the accuracy and topic relevance, as it is commonly done [2].…”

Section: Impact and Conclusionmentioning

confidence: 99%