Knowing the quality of reading comprehension (RC) datasets is important for the development of natural-language understanding systems. In this study, two classes of metrics were adopted for evaluating RC datasets: prerequisite skills and readability. We applied these classes to six existing datasets, including MCTest and SQuAD, and highlighted the characteristics of the datasets according to each metric and the correlation between the two classes. Our dataset analysis suggests that the readability of RC datasets does not directly affect the question difficulty and that it is possible to create an RC dataset that is easy to read but difficult to answer.

No abstract

An overview of the SemEval-2 Japanese WSD task is presented. The new characteristics of our task are (1) the task will use the first balanced Japanese sense-tagged corpus, and (2) the task will take into account not only the instances that have a sense in the given set but also the instances that have a sense that cannot be found in the set. It is a lexical sample task, and word senses are defined according to a Japanese dictionary, the Iwanami Kokugo Jiten. This dictionary and a training corpus were distributed to participants. The number of target words was 50, with 22 nouns, 23 verbs, and 5 adjectives. Fifty instances of each target word were provided, consisting of a total of 2,500 instances for the evaluation. Nine systems from four organizations participated in the task.

Japanese corpora annotated with predicate-argument structure (PAS) have been constructed as part of several research projects and these annotated corpora have significantly advanced the field of PAS analysis. However, according to an inter-annotator agreement study and qualitative analysis of the existing corpora, there is still a strong need for further improvement of the annotation guidelines of the corpora. To improve the quality of PAS annotation guidelines, we have collected and summarized the practical knowledge and a list of problematic issues concerning the task of the PAS annotation through discussions with researchers actively engaged in the construction of NAIST Text Corpus (NTC) and Kyoto Text Corpus (KTC), researchers concerned

No abstract