A Cautionary Note on the Reliability of the Online Survey Data: The Case of Wage Indicator

Abstract

We investigate the reliability of data from the Wage Indicator (WI), the largest online survey on earnings and working conditions. Comparing WI to nationally representative data sources for 17 countries reveals that participants of WI are not likely to have been representatively drawn from the respective populations. Previous literature has proposed to utilize weights based on inverse propensity scores, but this procedure was shown to leave reweighted WI samples different from the benchmark nationally representative data. We propose a novel procedure, building on covariate balancing propensity score, which achieves complete reweighting of the WI data, making it able to replicate the structure of nationally representative samples on observable characteristics. While rebalancing assures the match between WI and representative benchmark data sources, we show that the wage schedules remain different for a large group of countries. Using the example of a Mincerian wage regression, we find that in more than a third of the cases, our proposed novel reweighting assures that estimates obtained on WI data are not biased relative to nationally representative data. However, in the remaining 60 percent of the analyzed 95 data sets, systematic differences in the estimated coefficients of the Mincerian wage regression between WI and nationally representative data persist even after reweighting. We provide some intuition about the reasons behind these biases. Notably, objective factors such as access to the Internet or richness appear to matter, but self-selection (on unobservable characteristics) among WI participants appears to constitute an important source of bias.

Comparison Reviews on E-commerce Research Methods: Suggestions for Post-COVID-19

“…For example, surveys might be descriptive, collecting information about the sampled subjects or evaluating intervention outcomes involving openended or closed questions, rating questions, or ranking questions (Bargagliotti et al, 2021). On the other side of the discussion, opponents of Internet-based survey data questioned the quality of such data, response rate, or insufficient time to complete the survey (Smyk et al, 2021). Another limitation was that most large-scale survey programs could not consider specific requirements of subgroups of respondents by adapting questions or survey procedures (Schanze, 2021).…”

Methodological issues related to the use of online labour market data

“…The respondents of the survey are self-selected on a voluntary basis in response to the online campaign, which results in a biased dataset. There is a lively discussion among scholars as to what extent this bias can be sufficiently addressed using methods such as weighting (Fabo and Kahanec 2018;Smyk, Tyrowicz, and van der Velde 2018;Tijdens and Steinmetz 2016). In developed countries, online panels with probabilistic sampling have recently emerged as a promising means of collecting high-quality, reliable data (Das, Ester, and Kaczmirek 2018).…”

Effective recruitment or bot attack? The challenge of online research surveys and recommendations to reduce risk and improve robustness: a case study commentary (Preprint)

“…This increase may be linked to the rise of online paid research panels and crowdworker platforms such as Amazon Mechanical Turk, which bots can leverage. Whether the bots are automated malware or human respondents who are completing surveys for financial gain without meeting study inclusion criteria [8], it is irrefutable that caution must be exercised when conducting an online or remote study [9,10]. Bot attacks must be identified as they impact the integrity of datasets [5,11,12], creating a situation where ineffective interventions may appear effective, and conversely, effective interventions may seem ineffective.…”