Online and Social Media Data As an Imperfect Continuous Panel Survey

Abstract

There is a large body of research on utilizing online activity as a survey of political opinion to predict real world election outcomes. There is considerably less work, however, on using this data to understand topic-specific interest and opinion amongst the general population and specific demographic subgroups, as currently measured by relatively expensive surveys. Here we investigate this possibility by studying a full census of all Twitter activity during the 2012 election cycle along with the comprehensive search history of a large panel of Internet users during the same period, highlighting the challenges in interpreting online and social media activity as the results of a survey. As noted in existing work, the online population is a non-representative sample of the offline world (e.g., the U.S. voting population). We extend this work to show how demographic skew and user participation is non-stationary and difficult to predict over time. In addition, the nature of user contributions varies substantially around important events. Furthermore, we note subtle problems in mapping what people are sharing or consuming online to specific sentiment or opinion measures around a particular topic. We provide a framework, built around considering this data as an imperfect continuous panel survey, for addressing these issues so that meaningful insight about public interest and opinion can be reliably extracted from online and social media data.

Dissecting the Spirit of Gezi: Influence vs. Selection in the Occupy Gezi Movement

“…As we have repeatedly noted, Twitter data is potentially incomplete or biased in numerous respects (Diaz et al, 2015), and hence must be carefully preprocessed before any conclusions are drawn. Because this process necessarily relies on the existence of sources of "ground truth" data such as surveys, interviews, and electoral records, our approach is intrinsically complementary to traditional methods.…”

“…Critics of social media data, meanwhile, tend to focus on its quality, which is often subject to problems such as lack of representativeness or completeness (Tufekci, 2014), high variability over time and/or across individuals (Salganik and Levy, 2012), and absence of relevant outcome variables (Diaz et al, 2015). In constructing a panel out of what is effectively a convenience sample of Twitter users we have attempted to strike a balance between data quantity and quality, on the one hand leveraging the low cost and "always on" nature of Twitter data while on the other hand alleviating potential sources of error.…”

“…12 A similar method has been proposed by Diaz et al (2015) in the context of public opinion polling.…”

The empiricist’s challenge: Asking meaningful questions in political science in the age of big data

“…But if we want to use digital trace data to draw inferences on the general population, this systematic skewness becomes very problematic. This is reinforced as the composition of the active user base of digital services, such as Facebook and Twitter, for example, seems to fluctuate unforeseeably over time and in reaction to specific events (Diaz, Gamon, Hofman, Kiciman, & Rothschild, 2016). This makes the development of weighing procedures that would account for a stable skewness between the users of digital services and the general population unfeasible.…”

Education policy research in the big data era: Methodological frontiers, misconceptions, and challenges

“…Specifically, younger people are more likely than older people to use three popular social media sites-Facebook, Twitter, and LinkedIn; those with more education and higher income are more likely to use these three sites than those who have lower socioeconomic status; women are more likely to use Facebook but less likely to use LinkedIn; African Americans are more likely to be on LinkedIn and Twitter, but less likely to use Facebook than Whites (Hargittai, 2015). As a corollary, the online population demographics might shift constantly, and do not represent the ones in the offline world (Diaz, Gamon, Hofman, Kıcıman, & Rothschild, 2016).…”