Semi-automatic extraction of multiword terms from domain-specific corpora

Abstract

Purpose A hybrid approach is presented, which combines linguistic and statistical information to semi-automatically extract multiword term candidates from texts. Design/methodology/approach The method is designed to be domain and language independent, focusing on languages with rich morphology. Here, it is used for extracting multiword terms from texts in Serbian, belonging to the agricultural engineering domain, as a use case. Predefined syntactic structures were used for multiword terms. For each structure, a finite state transducer was developed, which recognizes text sequences having that structure and outputs the sequence in a normalized form, so that different inflectional forms of the same multiword term can be counted properly. Term candidates were further filtered by their frequencies and evaluated by two domain experts. Findings By using language resources, such as electronic dictionaries and grammars, 928 multiword terms were extracted out of 1,523 multiword terms that were recognized as candidates from a corpus having 42,260 different simple word forms; 870 of these were new, not already contained in the existing electronic dictionary of compounds for Serbian, and they were used to enrich the dictionary. Originality/value The paper presents methodology that can significantly contribute to the development of terminology lexicons in different areas. In this particular use case, some important agricultural engineering concepts were extracted from the text, but this approach could be used for other domains and languages as well.

A Novel Approach: Tokenization Framework based on Sentence Structure in Indonesian Language

“…Research in Serbian language with agricultural engineering domain conducted by [28] provides a hybrid approach by combining linguistic and statistical information. The Candidate terms are obtained using the frequency of occurrence of text sequences in the corpus.…”

HerCulB: content-based information extraction and retrieval for cultural heritage of the Balkans

“…In the case of the Serbian language, a lot of work has been done in the fields of IE and NERfrom general analysis of necessary resources and appropriate methods (Vitas and Pavlovi c-Lažeti c, 2008), over collections of handcrafted rules for specific types of named entities (Gucul-Milojevi c, 2010;Krstev et al, 2011;Paji c et al, 2011), to the extraction of multiword expressions and named entities in specific domains, for example, multiword expressions in the agricultural engineering domain, mining and geology, named entities in the culinary domain, weather information from meteorological texts or keyword-based search of bilingual digital libraries (Paji c et al, 2012(Paji c et al, ,2018Stankovi c et al, 2016;Stankovi c-Vuji ci c et al, 2014). Some of the initiatives for spoken archives that use NLP for IE include the CHoral project (Heeren et al, 2009), aimed at building technology for spoken document retrieval for heritage collections and New South Voices (Atkins Library, University of North Carolina at Charlotte), that provides access to transcripts of interviews, narratives and conversations documenting life in the 20th century.…”

Deep Learning Methods for Identification of Multiword Flower and Plant Names

“…Taking into consideration the quick development of NLP technologies and the importance of Machine translation (MT) in the dissemination of knowledge and in building new resources, it is important to extend the studies and cover new areas of research, such as Botany. The automatic identification of terms in this field helps in improving the quality of NLP applications, computer assisted translation tools and automatic translation tools (Temmerman and Knops, 2004) as well as lexicon creation, acquisition of novel terms, text classification, text indexing, machine-assisted translation and other NLP tasks (Pajić et al, 2018). For this reason, in this paper, we focus on the automatic extraction of flower and plant names, and we intend to address the shortcomings in this domain with the help of AI.…”

“…

Multiword Terms (MWTs) are domain-specific Multiword Expressions (MWE) (Pajić et al, 2018) where two or more lexemes converge to form a new unit of meaning (León Araúz and Cabezas García, 2020). The task of processing MWTs is crucial in many Natural Language Processing (NLP) applications, including Machine Translation (MT) and terminology extraction.

…”