Unrepresentative big surveys significantly overestimated US vaccine uptake

Bradley, Valerie C.; Kuriwaki, Shiro; Isakov, Michael; Sejdinović, Dino; Meng, Xiao‐Li; Flaxman, Seth

doi:10.1038/s41586-021-04198-4

Cited by 196 publications

(191 citation statements)

References 28 publications

Supporting

Mentioning

185

Contrasting

Order By: Relevance

“…Step 4: Bias adjustment: Since both fixed-and random-effects models were affected by the presence of very large datasets (albeit to a different extent), the question arises whether there is a possibility of adjusting for this large study bias. In Bradley et al (2021), the error could be directly deduced due to the nature of the survey data. This is however not easily possible in the context of meta-analysis.…”

Section: Discussionmentioning

confidence: 99%

“…Recent studies have demonstrated that the collection of extremely large datasets comes at the cost of data quality (Meng, 2018). A recent study has demonstrated that the effective sample size in large surveys can diminish by more than 99.99% if systematic bias is accounted for (Bradley et al, 2021). Since meta-analyses are typically based on inferential statistics and, even in random-effects models, weigh the effect sizes of individual datasets based on their respective sample size (although Bayesian models for meta-analysis also exist, Röver (2017)), this systematic bias can easily carry over to the meta-analytic findings.…”

Section: Introductionmentioning

confidence: 99%

“…Bootstrapping is routinely used to resample original data in order to achieve an underlying sampling distribution. Here, we use non-parametric bootstrapping on a given study sample previously used in meta-analysis to (1) test the robustness of the reported effect size by randomly in-and excluding datasets with replacement, (2) assess how much effects and confidence intervals change if less or more datasets were to be included (under the assumption that these additional datasets have comparable features to the already sampled datasets), (3) identify influential datasets based on big data more reliably than standard approaches, such as funnel plots, and (4) investigate if the bias induced by these influential datasets can be accounted for using a systematic adjustment of sample sizes (Bradley et al, 2021). These steps are critical for researchers conducting a meta-analysis to ascertain whether more datasets should be added to the current study sample or whether sample sizes need adjustment to account for bias.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Using bootstrapping to account for sampling variation and large study bias in meta-analysis

Schmitz¹,

Packheiser²,

Παπαδάτου-Παστού³

et al. 2022

Preprint

View full text Add to dashboard Cite

Meta-analysis is a commonly used tool to provide insight into effect estimates with high confidence due to the increase in power and resistance to sampling bias compared to individual studies. However, meta-analysis is not without flaws as bias from original research can easily carry over to the synthesized analysis. In the present study, we used non-parametric bootstrapping to address the issues of non-random sampling (i.e., the fact that meta-analyses aim to include all available data instead of random samples from the population), as well as large study bias in meta-analysis. We found that bootstrapping can be highly informative regarding (1) the robustness of effect estimates despite non-random sampling, (2) how the inclusion of additional datasets affects the effect estimate and its error, and (3) to identify influential datasets. We furthermore found that the standard deviation of the effect estimate decreases logarithmically with the addition of further datasets. Regarding large study bias, we found that (4) even in random-effects models, very large datasets have the potential to bias the data significantly. Using a bias adjustment method that replaces sample sizes of individual datasets by their square root radicand, we could account for the bias without losing a significant amount of information. Overall, we suggest that bootstrapping should be routinely used to estimate the robustness of effect estimates and assess large study bias in meta-analysis.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Using bootstrapping to account for sampling variation and large study bias in meta-analysis

Schmitz¹,

Packheiser²,

Παπαδάτου-Παστού³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Notably, the follow up response rate in the present study was relatively high at 81%. Thus, any potential variability can be predominantly due to recall bias of the respondents, which has been reported as substantial for unrepresentative surveys [17]. However, the phone survey used as follow up in the present study was performed by quali ed study personnel, and the participants were highly likely to answer the follow up questions accurately due to potential relevance of these questions to the parental microcirculation study.…”

Section: Limitationsmentioning

confidence: 98%

Detection of anti-SARS-CoV-2 S1 RBD-specific antibodies prior to and during the pandemic in 2011-2021 and Covid-19 observational study in 2019-2021

Gumanova¹,

Gorshkov²,

Bogdanova³

et al. 2022

Preprint

View full text Add to dashboard Cite

BackgroundLongitudinal surveys to continually monitor the seroprevalence around the globe are required to support the prevention and control efforts for assessment of the levels of endemic stability in certain countries and regions. We investigated seroprevalence of anti-SARS-CoV-2-S1 receptor-binding domain (RBD)–specific antibodies in the samples of participants recruited for various cardiovascular studies unrelated to exposure to the virus in 2011-2021. A cohort study recruited in 2019-2021 was used to evaluate the vaccination and anti-IgG-SARS-CoV-2–S1 RBD-positive statuses to assess the resistance and severity of Covid-19. Materials and MethodsThe present observational study included subjects 18 years of age and older randomly selected from various cohorts previously recruited from 2011 to 2021 from the city of Moscow and Moscow Region for a total of 565 serum samples including recent participants (N = 310); recruited in 2019-2021. The phone survey of the participants of this cohort was conducted in October, 2021. Anti-SARS-CoV-2 S1 RBD-specific antibodies were assayed in the serum samples (N = 565). At endpoint (October 30, 2021), participants of the 2019-2021 cohort (N = 310) were interviewed over phone using a questionnaire that included 5 questions as follows: 1) did the participant experience a respiratory disorder at any time before or after the enrolment in the study; 2) the date of the disorder; 3) severity of the disorder, including asymptomatic, mild, medium (fever and cough), or severe (hospitalization); 4) was the participant vaccinated and the date of vaccination; and 5) brand of a vaccine. ResultsThe data of the present study indicated that a small percentage (3-6%) of SARS-CoV-2-S1 RBD-specific antibodies was detected in participants recruited in 2011-2019. The percentage of SARS-CoV-2-S1 RBD-specific antibodies was increased to 16.5% in 2020 and to 46% in 2021. The vaccination rate of 238 respondents of this cohort was 58% from August, 2020 to October, 2021. If this rate is added to the rate of not vaccinated SARS-CoV-2-S1 RBD-positive subjects (11%), collective immunity was able to reach 69%. The data indicated that 12% of respondents were hospitalized. No cases of hospitalization were reported in vaccinated respondents. The morbidity rate in the subgroup of anti-SARS-CoV-2-S1 RBD-positive respondents was 5.4-fold higher than that in the subgroup of vaccinated respondents (OR = 5.4; CI: 1.4 -19.8; P = 0.01).ConclusionDetected small percentage of SARS-CoV-2-S1 RBD-specific antibodies in participants recruited in 2011-2019 indicated possible spreading of coronaviruses during the pre-pandemic period. Collective immunity in Moscow and Moscow region was able to reach 69% from August, 2020 to October, 2021. Obtained results demonstrated that vaccination protects against COVID-19 and related hospitalization

show abstract

“…No: 'big' does not equate to 'quality'; a large, unrepresentative sample will provide poorer inferences about a population than a small, highly representative sample. An attempt to test this has been made by Bradley et al 4 through defining a formal measure that links data quantity, quality and problem difficulty, applied to Covid vaccine uptake in US adults. They compared a well-designed household survey of around 1000 people per week, to two much larger opportunistic surveys from Delphi-Facebook (250 000 people per week) and Census Household Pulse (75 000 people every 2 weeks).…”

mentioning

confidence: 99%