Pulling Out the Stops: Rethinking Stopword Removal for Topic Models

Schofield, Alexandra; Magnusson, Måns; Mimno, David

doi:10.18653/v1/e17-2069

Cited by 143 publications

(74 citation statements)

References 4 publications

(1 reference statement)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As both can be prone to misspellings, one has to check the documents carefully before applying a topic model 13 For tools to carry out these steps, see Graham et al (2016). For a discussion of the effect of stopword-removal, see Schofield et al (2017).…”

Section: Topic Models In Practicementioning

confidence: 99%

Economic history goes digital: topic modeling the Journal of Economic History

Wehrheim

2018

Cliometrica

View full text Add to dashboard Cite

Digitization and computer science have established a whole new set of methods to analyze large collections of texts. One of these methods is particularly promising for economic historians: topic models, statistical algorithms that automatically infer themes from large collections of texts. In this article, I present an introduction to topic modeling and give a very first review on the research using topic models. I illustrate their capacity by applying them on 2.675 articles published in the Journal of Economic History between 1941 and 2016. This contributes to traditional research on the JEH and to current research on the cliometric revolution.JEL-Classification: A12, C18, N01 as well as the participants in the research seminar in economic and social history at University of Regensburg for invaluable advice. This paper was presented in the lecture series on digital humanities at University of Regensburg.

show abstract

Section: Topic Models In Practicementioning

confidence: 99%

Economic history goes digital: topic modeling the Journal of Economic History

Wehrheim

2018

Cliometrica

View full text Add to dashboard Cite

show abstract

“…For our research purposes, LDA analysis using all the vocabulary words would not be appropriate, because the existence of many common or stop words often introduces meaningless or uninterpretable topics (Schofield et al 2017). Therefore, in this experiment, to remove common or stop words, the instructor and the tutor selected 382 representative words of the course.…”

Section: Corpus Of Reportsmentioning

confidence: 99%

Diverse reports recommendation system based on latent Dirichlet allocation

et al. 2017

View full text Add to dashboard Cite

This paper presents a proposal for system supporting learners in improving their report-writing skills by recommending reports from previous learners. The proposed system recommends reports that share similar subjects but which have different structures, expressions, and originality based on the distributions of words and subjects within the reports, as estimated using latent Dirichlet allocation (LDA). An important assumption made for this study is that reports with different word distributions tend to include different structures, expressions, and originality when they share similar subjects. Based on that assumption, the system selects and recommends reports that have dissimilar word distributions but which share similar subject distributions with a learner's report. The proposed system is expected to enhance learning of various writing skills from other learners. Finally, this paper demonstrates the effectiveness of the proposed system through actual data experiments.

show abstract

“…The LDA algorithm exploits documental-level word co-occurrence patterns to discover underlying topics. Based on a prior study, we first removed stop words (e.g., "the", "a") and words that occurred £3 times in our corpus [25].…”

Section: Step 3: Topic Modelingmentioning

confidence: 99%

Mining Twitter to assess the determinants of health behavior toward human papillomavirus vaccination in the United States

Zhang

Wheldon

Dunn

et al. 2019

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

ObjectivesTo test the feasibility of using Twitter data to assess determinants of consumers' health behavior towards Human papillomavirus (HPV) vaccination informed by the Integrated Behavior Model (IBM). MethodsWe used three Twitter datasets spanning from 2014 to 2018. We preprocessed and geocoded the tweets, and then built a rule-based model that classified each tweet into either promotional information or consumers' discussions. We applied topic modeling to discover major themes, and subsequently explored the associations between the topics learned from consumers' discussions and the responses of HPV-related questions in the Health Information National Trends Survey (HINTS). ResultsWe collected 2,846,495 tweets and analyzed 335,681 geocoded tweets. Through topic modeling, we identified 122 high-quality topics. The most discussed consumer topic is "cervical cancer screening"; while in promotional tweets, the most popular topic is to increase awareness of "HPV causes cancer". 87 out of the 122 topics are correlated between promotional information and consumers' discussions. Guided by IBM, we examined the alignment between our Twitter findings and the results obtained from HINTS. 35 topics can be mapped to HINTS questions by keywords, 112 topics can be mapped to IBM constructs, and 45 topics have statistically significant correlations with HINTS responses in terms of geographic distributions. ConclusionNot only mining Twitter to assess consumers' health behaviors can obtain results comparable to surveys but can yield additional insights via a theory-driven approach. Limitations exist; nevertheless, these encouraging results impel us to develop innovative ways of leveraging social media in the changing health communication landscape. BACKGROUND and SIGNIFICANCEHuman papillomavirus (HPV) is the most common sexually transmitted disease (STD) in the United States (US) [1]. Although HPV infections are transient, persistent infection can lead to cancer. An estimated 33,700 new patients are diagnosed with HPV-associated cancers (e.g., anal, penile, cervical, and oral cancers) each year [2] in US. HPV vaccine is effective in preventing most of these HPV-related cancers for individuals in early age [3]. Nevertheless, in 2017, only 48.6% of US adolescents received recommended HPV vaccination series, and 65.5% received ≥1dose of the series [4]. HPV vaccination coverage also varies greatly by state. Only three states (i.e., District of Columbia: 91.9%, Rhode Island: 88.6%, and Massachusetts: 81.9%) have more than 80% coverage for the first dose, while the bottom three states (i.e., Kentucky: 49.6%, and Mississippi: 49.6%, Wyoming: 46.9%) have coverage rates less than 50% [4]. There is a huge public health needs to increase the awareness of HPV-related issues to promote HPV vaccination.To increase HPV vaccination initiation and coverage, we first need to understand factors that affect people's health behavior towards vaccination uptake. Recognized by the Integrated

show abstract

Pulling Out the Stops: Rethinking Stopword Removal for Topic Models

Cited by 143 publications

References 4 publications

Economic history goes digital: topic modeling the Journal of Economic History

Economic history goes digital: topic modeling the Journal of Economic History

Diverse reports recommendation system based on latent Dirichlet allocation

Mining Twitter to assess the determinants of health behavior toward human papillomavirus vaccination in the United States

Contact Info

Product

Resources

About