Automatic syntactic analysis of free text

Abstract

Problems encountered under the syntactic analysis of free text documents are discussed. As these problems are of general interest in nonautomatic or automatic syntactic indexing or search systems, these problems are presented in detail. The system called COPSY (context operator syntax), which uses natural language processing techniques during fully automatic syntactic analysis (indexing and search) of free text documents is described. Applications under real world conditions are mentioned as well as evaluation and technical aspects. Further developments in the field of thesaurus building and full-text analysis using the linguistic algorithms of the syntactic retrieval system are outlined. COPSY was developed as part of a text processing project at Siemens, called TINA (Text-/n halts-Analyse: text content analysis). Software from TINA is actually being tested by the U.S. Department of Commerce for patent search and indexing.Traditional Techniques "Precision-oriented devices involve using term phrases instead of single terms" (Salton, 1986) and their development is being pursued more and more intensely (Doszkocs, 1986). The crucial issues that term combination (i.e., syntactic indexing and search) deals with are fundamental problems in information retrieval. Therefore, we will begin by discussing some general problems that arise in this context, in particular postcoordination and precoordination. PostcoordinationTo simulate syntactic relations between search terms, commercial information retrieval systems use context operators in order to get better results with regard to precision. Apart from this technique there is the possibility of building more sophisticated inverted files that invert free text search terms in "narrower" units than documents or even paragraphs: the inversion is organized on the sentence level. In this case, the Boolean AND search looks for terms within sentence boundaries using the operator often called "WITH." This technique, too, can be regarded as a Received June 9, 1987; revised March 3, 1988; accepted July 11. 1988. C 1990 very crude means of doing a kind of pseudo-syntax by grouping terms close together.These well known techniques allow a post-coordinate search which combines search terms on-line. The advantage of searching in this manner is that the producers of information retrieval software do not really have to offer new systems. Instead they need only integrate more complex inverted files with more specific search sections (paragraphs, sentences) where search terms can be found using Boolean AND. Integrating word position numbers in context operator searches (often called "ADJACENT" or, more general, "NEAR") on the other hand does not imply a new concept in inverted file constructions.The disadvantages of the above mentioned search facilities are well known to searchers: complicated search logics have to be built up by the searcher and he or she is more or less forced to predict formulations his or her search question might have in the free text documents.If the searcher is lookin...

Partial parsing via finite-state cascades

“…), Cass2: 1300-2300 w/s, Copsy: 2700? w/s [11]). Cass2 is as fast as any parser in this class, with the possible exception of Copsy, for which the hardware adjustment is highly uncertain.…”

Standardization of Terms Applying Finite-State Transducers (FST)

“…In addition to the surface analysis and the analysis of fragments from the corpus, many systems effectuate a POS category disambiguation process (Kupiec, 1992). The syntactic variants identified through these methods can be grouped, finally, in canonical syntactic structures (Schwarz, 1990;Sheridan & Smeaton, 1992;Smadja, 1993;Strzalkowski, 1996).…”

Semantic relations in information science

“…The retrieval results from syntactic relation matching appears to be no better than the results obtainable using index phrases generated using statistical methods, such as those described by Fagan (1989). Metzler & Haas (1989), Metzler, Haas, Cosic & Weise (1990), Schwarz (1990), and Ruge, Schwarz, & Warner (1991) performed syntactic processing to produce dependency trees that indicate which terms modify which other terms. Smeaton & van Rijsbergen (1988) found that the premodifier-headnoun relation (e.g., adjective-noun) has a bigger impact on retrieval than other relations.…”