Storywrangler: A massive exploratorium for sociolinguistic, cultural, socioeconomic, and political timelines using Twitter

Abstract

In real time, Twitter strongly imprints world events, popular culture, and the day-to-day, recording an ever-growing compendium of language change. Vitally, and absent from many standard corpora such as books and news archives, Twitter also encodes popularity and spreading through retweets. Here, we describe Storywrangler, an ongoing curation of over 100 billion tweets containing 1 trillion 1-grams from 2008 to 2021. For each day, we break tweets into 1-, 2-, and 3-grams across 100+ languages, generating frequencies for words, hashtags, handles, numerals, symbols, and emojis. We make the dataset available through an interactive time series viewer and as downloadable time series and daily distributions. Although Storywrangler leverages Twitter data, our method of tracking dynamic changes in n-grams can be extended to any temporally evolving corpus. Illustrating the instrument’s potential, we present example use cases including social amplification, the sociotechnical dynamics of famous individuals, box office success, and social unrest.

Say Their Names: Resurgence in the collective attention toward Black victims of fatal police violence following the death of George Floyd

“…We use the first and last name of each victim in our dataset to create a set of two word phrases, or 2-grams. For each name, we query its frequency and rank over time using the Storywrangler API 2 [32]. Storywrangler uses a 10% sample of English tweets to measure how often words and phrases (also known generally as ngrams 3 ) are used on Twitter.…”

“…The long-term increase in the rate of retweets is confounded by other factors though, such as changes to Twitter's design and algorithmic curation. To account for this, we define the relative social amplifica-tion [32,36] as…”

Computational timeline reconstruction of the stories surrounding Trump: Story turbulence, narrative control, and collective chronopathy

“…We draw on a collection of around 10% of all tweets starting in 2008. We take all English language tweets [ 35 , 36 ] matching the word ‘Trump’ from 2015/01/01 on. We ignore case and accept matches of ‘Trump’ at any location of a tweet (e.g., ‘@RealDonaldTrump’ matches).…”

Augmenting Semantic Lexicons Using Word Embeddings and Transfer Learning

“…Notably, language is an evolving sociotechnical phenomenon. New words and phrases are created constantly, especially on social media (Alshaabi et al, 2021a ). Word usage changes over time.…”