The NeoCrawler: identifying and retrieving neologisms from the internet and monitoring ongoing change

Abstract

Why do some new words manage to enter the lexicon and stay there while others drop out of use and are neither used nor heard anymore? Of interest to both lay people and linguists, this question has not been answered in an empirically convincing manner to date, mainly because systematic methods have not yet been found for spotting new words as soon as possible after their first occurrence and monitoring their early development and spread as exhaustively as possible. In this paper we present a new and improved tool which is designed to accomplish precisely these tasks when applied to material from the Internet. Following a brief review of existing tools for retrieving linguistic data from the Web (Section 2), we will introduce in some detail a tailor-made webcrawler, the so-called NeoCrawler, which identifies and retrieves neologisms from the Internet and stores data necessary for the systematic monitoring of their early development with regard to form and meaning as well as spread (Section 3). Following this description, we will present a case study discussing the results of an analysis of the neologism detweet with regard to its di¤usion, institutionalization, lexicalization and lexical networkformation (Section 4). The study indicates that the NeoCrawler can indeed be applied fruitfully in the study of ongoing processes relating to how the meanings and forms of new words are negotiated in the speech community, how words spread in the early stages of their life cycles and how they begin to establish themselves in lexical and semantic networks. (V9 8/9/11 18:34) WDG (155mmÂ230mm) TimesNRMT 1317 Allan pp. 59-96 1317 Allan_04_Kerremans (p. 61) The NeoCrawler 61 6. RSS and Atom feeds are tools that enable users to update, publish and exchange web content easily. They contain basic information about the content, such as title, link, description and publication date in XML format. GlossaNet 2 uses this link to access and download the page into the corpus. 7. To our current knowledge, the LSE has not been realized (yet).

Mapping Lexical Innovation on American Social Media

“…More recently, researchers have focused on how words become lexicalized as they gradually settle in on particular forms and meanings and how words become institutionalized as they enter into the standard vocabulary of a language (Brinton & Traugott 2005). Lexicographers have also devoted considerable effort to identifying and defining neologisms, including through the analysis of newspaper corpora (Baayen & Renouf 1996) and internet search engine results (Kerremans et al 2011). Although there has been sustained interest from linguists on the formation and development of new words, regional patterns of lexical innovation-both in terms of their origin and spread-have been left largely unexplored.…”

Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics

“…Further filters then apply to eliminate spellings errors and Proper Nouns. Subsequent developments all replicate this architecture : OBNEO (Cabré and De Yzaguirre, 1995), NeoCrawler (Kerremans et al, 2012), Logoscope (Gérard et al, 2014) and more recently Neoveille (Cartier, 2016). Four main difficulties arise from these architecture : first, EDA can not track semantic neologisms, as they use existing lexical units to convey innovative meanings; second, the design of a reference exclusion dictionary is not that obvious as it requires the existence of a machinereadable dictionary : this entails specific procedures to apply this architecture to less-resourced languages, and the availability of an up-to-date machine-readable dictionary for more resourced languages ; third, the EDA architecture is not sufficient in itself : among unknown forms, most of them are Proper Nouns, spelling mistakes and other cases derived from corpus boilerplate removal : this entails a post-processing phase to depart cases; Fourth, these systems do not take into account the sociological and diatopic aspects of neologism, as they limit their corpora to specific domains : a ideal system should be able to extend its monitoring to new corpora and maintain diastratic meta-datas to characterize novel forms.…”

Wortschatzwandel im „Spiegel“

“…Diachrone Wortschatzveränderungen werden in der Regel exemplarisch anhand bestimmter Phänomene oder Phänomenbereiche untersucht. Zum Beispiel analysiert man, welche lexikalischen Innovationen im Zeitverlauf zu beobachten sind (Kerremans/Stegmayr/Schmid 2012;Würschinger et al 2016; Müller-Spitzer/Wolfer/Kolpenig 2018). Unter einer quantitativen Perspektive wird dabei nicht nach neuen Wörtern im linguistischen Sinn gesucht, sondern entweder nach neuen Token oder neuen automatisch lemmatisierten Worttypen, die in einem neuen Jahr, einem neuen Monat oder einer neuen Ausgabe auftauchen.…”