Improving the Reproducibility of PAN’s Shared Tasks:

Abstract

Potthast, M.; Gollub, T.; Rangel, F.; Rosso, P.; Stamatatos, E.; Abstract This paper reports on the PAN 2014 evaluation lab which hosts three shared tasks on plagiarism detection, author identification, and author profiling.To improve the reproducibility of shared tasks in general, and PAN's tasks in particular, the Webis group developed a new web service called TIRA, which facilitates software submissions. Unlike many other labs, PAN asks participants to submit running softwares instead of their run output. To deal with the organizational overhead involved in handling software submissions, the TIRA experimentation platform helps to significantly reduce the workload for both participants and organizers, whereas the submitted softwares are kept in a running state. This year, we addressed the matter of responsibility of successful execution of submitted softwares in order to put participants back in charge of executing their software at our site. In sum, 57 softwares have been submitted to our lab; together with the 58 software submissions of last year, this forms the largest collection of softwares for our three tasks to date, all of which are readily available for further analysis. The report concludes with a brief summary of each task.

Corpus Selection Approaches for Multilingual Parsing from Raw Text to Universal Dependencies

“…For the submitted official results through TIRA (Potthast et al, 2014), we use the intersection model for all languages. Since we focus on the corpus selection, we do not perform additional preprocessing and we use the provided training datasets as they are.…”

CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies

“…Key goals of any empirical evaluation are to ensure a blind evaluation, its replicability, and its reproducibility. To facilitate these goals, we employed the cloud-based evaluation platform TIRA (Potthast et al, 2014), 13 which implements the evaluation as a service paradigm (Hanbury et al, 2015). In doing so, we depart from the traditional submission of system output to shared tasks, which lacks in these regards, toward the submission of working software.…”

A non-projective greedy dependency parser with bidirectional LSTMs

“…All of these test sets (Nivre et al, 2017b) were hidden from the participating teams until the shared task had ended. Using the TIRA environment (Potthast et al, 2014) provided for the shared task, participants could execute runs on them, but not see the outputs or the results.…”