Living Labs for Academic Search at CLEF 2020

Schaer, Philipp; Schaible, Johann; Müller, Burkhard

doi:10.1007/978-3-030-45442-5_75

Cited by 4 publications

(4 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As initially mentioned, intermediate evaluation in a living lab setting is especially challenging because of the lacking test collection. To provide a starting point for the LiLAS lab, Schaer et al provided head queries and candidate documents from the two real-world academic search systems, allowing the construction of pseudo test collections [21]. Pseudo test collections are a long-established method to create synthetic queries, and relevance judgments [6].…”

Section: Related Workmentioning

confidence: 99%

Evaluating Research Dataset Recommendations in a Living Lab

Keller

Munz

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The search for research datasets is as important as laborious. Due to the importance of the choice of research data in further research, this decision must be made carefully. Additionally, because of the growing amounts of data in almost all areas, research data is already a central artifact in empirical sciences. Consequentially, research dataset recommendations can beneficially supplement scientific publication searches. We formulated the recommendation task as a retrieval problem by focussing on broad similarities between research datasets and scientific publications. In a multistage approach, initial recommendations were retrieved by the BM25 ranking function and dynamic queries. Subsequently, the initial ranking was re-ranked utilizing click feedback and document embeddings. The proposed system was evaluated live on real user interaction data using the STELLA infrastructure in the LiLAS Lab at CLEF 2021. Our experimental system could efficiently be fine-tuned before the live evaluation by pre-testing the system with a pseudo test collection based on prior user interaction data from the live system. The results indicate that the experimental system outperforms the other participating systems.

show abstract

Section: Related Workmentioning

confidence: 99%

Evaluating Research Dataset Recommendations in a Living Lab

Keller

Munz

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…It is integrated in these online academic search engines, which enables participants to evaluate their experimental retrieval systems (e.g., domainspecific research data recommendations based on publications [26]) with real users. STELLA will be part of the Living Labs for Academic Search (LiLAS) lab at CLEF 2020 [25] 10 .…”

Section: A Novel Concept For Evaluation Infrastructuresmentioning

confidence: 99%

Evaluation Infrastructures for Academic Shared Tasks

et al. 2020

Self Cite

View full text Add to dashboard Cite

Academic search systems aid users in finding information covering specific topics of scientific interest and have evolved from early catalog-based library systems to modern web-scale systems. However, evaluating the performance of the underlying retrieval approaches remains a challenge. An increasing amount of requirements for producing accurate retrieval results have to be considered, e.g., close integration of the system's users. Due to these requirements, small to mid-size academic search systems cannot evaluate their retrieval system in-house. Evaluation infrastructures for shared tasks alleviate this situation. They allow researchers to experiment with retrieval approaches in specific search and recommendation scenarios without building their own infrastructure. In this paper, we elaborate on the benefits and shortcomings of four state-of-the-art evaluation infrastructures on search and recommendation tasks concerning the following requirements: support for online and offline evaluations, domain specificity of shared tasks, and reproducibility of experiments and results. In addition, we introduce an evaluation infrastructure concept design aiming at reducing the shortcomings in shared tasks for search and recommender systems.

show abstract

“…After LiLAS ran as a workshop lab at CLEF 2020 [12,13], in 2021 a full evaluation lab will take place. This lab's unique selling point is that we offer two tasks to test this approach in two different academic search domains and evaluation setups.…”

Section: Introductionmentioning

confidence: 99%

Living Lab Evaluation for Life and Social Sciences Search Platforms - LiLAS at CLEF 2021

Schaer

Schaible

Castro

2021

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Meta-evaluation studies of system performances in controlled offline evaluation campaigns, like TREC and CLEF, show a need for innovation in evaluating IR-systems. The field of academic search is no exception to this. This might be related to the fact that relevance in academic search is multilayered and therefore the aspect of user-centric evaluation is becoming more and more important. The Living Labs for Academic Search (LiLAS) lab aims to strengthen the concept of usercentric living labs for the domain of academic search by allowing participants to evaluate their retrieval approaches in two real-world academic search systems from the life sciences and the social sciences. To this end, we provide participants with metadata on the systems' content as well as candidate lists with the task to rank the most relevant candidate to the top. Using the STELLA-infrastructure, we allow participants to easily integrate their approaches into the real-world systems and provide the possibility to compare different approaches at the same time.

show abstract

Living Labs for Academic Search at CLEF 2020

Cited by 4 publications

References 7 publications

Evaluating Research Dataset Recommendations in a Living Lab

Evaluating Research Dataset Recommendations in a Living Lab

Evaluation Infrastructures for Academic Shared Tasks

Living Lab Evaluation for Life and Social Sciences Search Platforms - LiLAS at CLEF 2021

Contact Info

Product

Resources

About