3.5K runs, 5K topics, 3M assessments and 70M measures: What trends in 10 years of Adhoc-ish CLEF?

An Information Visualization Tool for the Interactive Component-Based Evaluation of Search Engines

“…the CLAIRE tool [1], see Fig. 1 -exists for dealing with large sets of IR systems -the GoP [6,7] -generated by many IR components which allows the inspection of both configurations and measures. CLAIRE is based on a totally different visual paradigm since it uses tiles, parallel coordinates and boxplots to explore system configurations.…”

An InfoVis Tool for Interactive Component-Based Evaluation

“…However, to the best of our knowledge only one solution -i.e. the CLAIRE tool [3], see Figure 7 -exists for dealing with large sets of IR systems -the GoP [6,7] -generated by many IR components which allows the inspection of both configurations and measures. CLAIRE is based on a totally different visual paradigm since it uses tiles, parallel coordinates and boxplots to explore system configurations.…”

Report on CLEF 2018

“…CLEF was established almost 20 years ago with a specific focus on stimulating research and innovation in multimodal and multilingual information access and retrieval. Over the years CLEF has fostered the creation of language resources in many European and non-European languages, promoted the growth of a vibrant and multidisciplinary research community, provided sizable improvements in the performance of monolingual, bilingual, and multilingual information access systems [5], and achieved a substantial scholarly impact [16,17] In its first 10 years, CLEF hosted a series of experimental labs that reported their results at an annual workshop held in conjunction with the European Conference on Digital Libraries (ECDL). In 2010, now a mature and well-respected evaluation forum, CLEF expanded to include a complementary peer-reviewed conference for discussion of advancing evaluation methodologies and reporting the evaluation of information access and retrieval systems regardless of data type, format, language, etc.…”