A diagnostic tool for German syntax

Abstract

In this pa per we describe an effort to construct a catalogue of syntacti c da ta, exemplifying the major syntactic patterns of German. The purpose of the corpus is to support t he diagnosis of errors in the syntact ic components of natural language processing (NLP) systems. Two secondary aims are the evaluation of NLP systems components and the support of theoretical and empirical work on German syntax. Th e data consist of artificially and systematically constructed expressions, including also negative (ungrammatical) examples. The data are organized into a relational data base and annotated with some basic information about the phenomena illustrated and the internal structure of the sample sentences . The organization of the data supports selected systematic testing of specifi c areas of syntax, but a lso serves the purpose of a linguistic data base. Th e paper first gives some general motivation for the necessity of syntactic precision in some areas of NLP and discusses the potential contri bution of a syntactic data base to the field of component evaluation . The second part of the paper describes the set up and control methods applied in the construct ion of t he sentence sui te and annota tions to the examples. We illustrate the approach with the examp le of verbal government. T he section also contains a descri ption of the abstract data model, the design of the data base and t he query language used to access the data. The final sections compare our work to existing approaches and sketch some future extensions.We invite other research groups to participate in our effort, so that the diagnostics tool can eventually become pub lic domain .

Towards systematic grammar profiling.Test suite technology 10 years after

“…systematic collections of artificially constructed and manually annotated reference data) have long been acknowledged as suitable for fine-grained diagnosis, progress evaluation, and benchmarking (see e.g. Flickinger, Nerbonne, Sag & Wassow, 1987;Nerbonne, Netter, Diagne, Dickmann & Klein, 1993; and Sparck Jones & Galliers, 1995). Most of the available data sets, however, follow the traditional design as flat text files listing test sentences annotated with, if at all, grammaticality judgements plus, in some cases, informal section headings grouping sets of sentences according to linguistic phenomena (or sometimes application-specific criteria).…”

Survey of the State of the Art in Human Language Technology

“…A large, mature test suite will be structured into a number of dimensions of elementary phenomena and contexts, and may include invalid as well as valid inputs, tagged as such. Nerbonne, Netter, et al (1993) describes a recent state-of-the-art example of this.…”

Natural language dialogue service for appointment scheduling agents

“…The e-mails were manually analyzed and annotated with major syntactic and semantic features as well as speechact information. A combination of two relational database systems was employed to ease the storage, maintenance, extension and retrieval of the NL data: (i) DiTo (Nerbonne et al, 1993), a full text database where the e-mails can be accessed, (ii) tsdb (Oepen et al, 1995), an elaborated fact database which permits the extraction of specific linguistic constructions together with the associated linguistic annotations. The annotation work is based on the TSNLP framework (Lehmann et al, 1996) where detailed category and function lists are defined for the structural and dependency structure annotation of linguistic material for NLP test suites.…”