Know thy corpus! Robust methods for digital curation of Web corpora

Abstract

This paper proposes a novel framework for digital curation of Web corpora in order to provide robust estimation of their parameters, such as their composition and the lexicon. In recent years language models pre-trained on large corpora emerged as clear winners in numerous NLP tasks, but no proper analysis of the corpora which led to their success has been conducted. The paper presents a procedure for robust frequency estimation, which helps in establishing the core lexicon for a given corpus, as well as a procedure for estimating the corpus composition via unsupervised topic models and via supervised genre classification of Web pages. The results of the digital curation study applied to several Web-derived corpora demonstrate their considerable differences. First, this concerns different frequency bursts which impact the core lexicon obtained from each corpus. Second, this concerns the kinds of texts they contain. For example, OpenWebText contains considerably more topical news and political argumentation in comparison to ukWac or Wikipedia. The tools and the results of analysis have been released.

Multimodal Pipeline for Collection of Misinformation Data from Telegram

“…Additionally, we calculated the frequency for each word in our only including English Telegram messages. Then, we used the frequency list of Open-WebTex (Sharoff, 2020) as a reference corpus for computing the log-likelihood (LL). This metric helps to identify the most indicative words in our corpus when compared against the reference.…”

ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Review

“…A popular choice that meets the aforementioned requirement is NVIDIA A100 GPU. According to Google Cloud Service Provider, 4 the hourly price of NVIDIA A100 GPU is around 3.93$. Now the information provided in [27,35] the training cost can be estimated as 21 days × 24 h × 3.93 USD × 2048 GPUs would yield 4.057 million dollars.…”

“…1.5 billion and 175 billion, and the pre-training data, i.e. WebText [4] and CommonCrawl [5], respectively. CommonCrawl is 15 × larger than WebText.…”

“…com/ conte nt/ dam/ en-zz/ Solut ions/ Data-Center/ a100/ pdf/ a100-80gb-datas heet-update-nvidia-us-15210 51-r2-web. pdf4 https:// cloud. google.…”