Comparing water quality valuation across probability and non‐probability samples

Abstract

We compare water quality valuation results from a probability sample and two opt‐in non‐probability samples, MTurk and Qualtrics. The samples differ in some key demographics, but measured attitudes are strikingly similar. For valuation models, most parameters were significantly different across samples, yet many of the marginal willingness to pay were similar across samples. Notably, for non‐marginal changes there were some differences by samples: MTurk values were always significantly greater than the probability sample, as were Qualtrics values for changes up to about a 20% improvement. Overall, the evidence is mixed, with some key differences but many similarities across samples.

Estimating the benefits to Florida households from avoiding another Gulf oil spill using the contingent valuation method: Internal validity tests with probability‐based and opt‐in samples

“…Probability-based internet samples are more expensive but respondents may pay more attention to the surveys and may generate higher-quality data. Recently, Sandstrom et al (2023) compared two opt-in panels, MTurk and Qualtrics, with a mixed-mode mail/internet sample. They found that each sample produced expected results but there were differences in the survey responses; MTurk respondents were more likely to vote in favor of the policy at each cost amount, which led to differences in WTP estimates.…”

Battling bots: Experiences and strategies to mitigate fraudulent responses in online surveys

“…Similarly, many researchers have taken advantage of conducting online surveys by purchasing responses through various marketing panels or platforms, for example, Qualtrics, Amazon's Mechanical Turk (MTurk), or Google Surveys. However, online surveys more acutely face issues affecting data quality, such as inattention or carelessness of participants (Cheng et al, 2022; Gao et al, 2016; Malone & Lusk, 2018a), nonrepresentative samples (Penn et al, 2023; Sandstrom et al, 2023; Whitehead et al, 2023) and fraudulent responses (Belliveau & Yakovenko, 2022; Chmielewski & Kucker, 2020; Griffin et al, 2021; Kramer et al, 2014; Rommel et al, 2022; Storozuk et al, 2020; Teitcher et al, 2015).…”

Hypothetical bias mitigation in representative and convenience samples

“…While a central concern of convenience samples is for online opt‐in panels versus probability‐based samples (Johnston, 2021, Sandstrom et al, 2023, Whitehead et al, 2023), we compare a volunteer convenience (snowball) sample recruited through broad advertisement via related organizations versus a probability‐based sample recruited via GfK (hereafter referred to as the representative sample), a well‐known market research firm. Our study context is to understand the WTP for beach condition information (BCI), comparing the results of a representative sample from a premier online sampling firm versus a convenience snowball sample.…”