Accuracy of four lateral flow immunoassays for anti SARS-CoV-2 antibodies: a head-to-head comparative study

Abstract

BackgroundSARS-CoV-2 antibody tests are used for population surveillance and might have a future role in individual risk assessment. Lateral flow immunoassays (LFIAs) can deliver results rapidly and at scale, but have widely varying accuracy.MethodsIn a laboratory setting, we performed head-to-head comparisons of four LFIAs: the Rapid Test Consortium’s AbC-19™ Rapid Test, OrientGene COVID IgG/IgM Rapid Test Cassette, SureScreen COVID-19 Rapid Test Cassette, and Biomerica COVID-19 IgG/IgM Rapid Test. We analysed blood samples from 2,847 key workers and 1,995 pre-pandemic blood donors with all four devices.FindingsWe observed a clear trade-off between sensitivity and specificity: the IgG band of the SureScreen device and the AbC-19™ device had higher specificities but OrientGene and Biomerica higher sensitivities. Based on analysis of pre-pandemic samples, SureScreen IgG band had the highest specificity (98.9%, 95% confidence interval 98.3 to 99.3%), which translated to the highest positive predictive value across any pre-test probability: for example, 95.1% (95%CI 92.6, 96.8%) at 20% pre-test probability. All four devices showed higher sensitivity at higher antibody concentrations (“spectrum effects”), but the extent of this varied by device.InterpretationThe estimates of sensitivity and specificity can be used to adjust for test error rates when using these devices to estimate the prevalence of antibody. If tests were used to determine whether an individual has SARS-CoV-2 antibodies, in an example scenario in which 20% of individuals have antibodies we estimate around 5% of positive results on the most specific device would be false positives.FundingPublic Health England.Research in contextEvidence before this studyWe searched for evidence on the accuracy of the four devices compared in this study: OrientGene COVID IgG/IgM Rapid Test Cassette, SureScreen COVID-19™ Rapid Test Cassette, Biomerica COVID-19 IgG/IgM Rapid Test and the UK Rapid Test Consortium’s AbC-19™ Rapid Test. We searched Ovid MEDLINE (In-Process & Other Non-Indexed Citations and Daily), PubMed, MedRxiv/BioRxiv and Google Scholar from January 2020 to 16th January 2021. Search terms included device names AND ((SARS-CoV-2) OR (covid)). Of 303 records assessed, data were extracted from 24 studies: 18 reporting on the accuracy of the OrientGene device, 7 SureScreen, 2 AbC-19™ and 1 Biomerica. Only three studies compared the accuracy of two or more of the four devices. With the exception of our previous report on the accuracy of the AbC-19™ device, which the current manuscript builds upon, sample size ranged from 7 to 684. For details, see Supplementary Materials.The largest study compared OrientGene, SureScreen and Biomerica. SureScreen was estimated to have the highest specificity (99.8%, 95% CI 98.9 to 100%) and OrientGene the highest sensitivity (92.6%), but with uncertainty about the latter result due to small sample sizes. The other two comparative studies were small (n = 65, n = 67) and therefore provide very uncertain results.We previously observed spectrum effects for the AbC-19™ device, such that sensitivity is upwardly biased if estimated only from PCR-confirmed cases. The vast majority of previous studies estimated sensitivity in this way.Added value of this studyWe performed a large scale (n = 4,842), head-to-head laboratory-based evaluation and comparison of four lateral flow devices, which were selected for evaluation by the UK Department of Health and Social Care’s New Tests Advisory Group, on the basis of a survey of test and performance data available. We evaluated the performance of diagnosis based on both IgG and IgM bands, and the IgG band alone. We found a clear trade-off between sensitivity and specificity across devices, with the SureScreen and AbC-19™ devices being more specific and OrientGene and Biomerica more sensitive. Based on analysis of 1,995 pre-pandemic blood samples, we are 99% confident that SureScreen (IgG band reading) has the highest specificity of the four devices (98.9%, 95% CI 98.3, 99.3%).We found evidence that all four devices have reduced sensitivity at lower antibody indices, i.e. spectrum effects. However, the extent of this varies by device and appears to be less for other devices than for AbC-19.Our estimates of sensitivity and specificity are likely to be higher than would be observed in real use of these devices, as they were based on majority readings of three trained laboratory personnel.Implications of all the available evidenceWhen used in epidemiological studies of antibody prevalence, the estimates of sensitivity and specificity provided in this study can be used to adjust for test errors. Increased precision in error rates will translate to increased precision in seroprevalence estimates. If lateral flow devices were used for individual risk assessment, devices with maximum specificity would be preferable. However, if, for an example, 20% of the tested population had antibodies, we estimate that around 1 in 20 positive results on the most specific device would be incorrect.

Evaluation of isotype specific salivary antibody assays for detecting previous SARS-CoV-2 infection in children and adults

“…The lowest observed specificity was for spike IgG in 30 to 39-year-olds (81.8%). There was no indication of specificity varying by sex for N-protein or spike (Table S8), which has been reported previously in other test accuracy studies 17,18 . In general, sensitivity declined with increasing time since symptom onset for all assays.…”

Evaluation of the Roche SARS-CoV-2 Rapid Antibody Test in samples from vaccinated individuals

“…In addition, the Elecsys Anti-SARS-CoV-2 S assay preferentially detects high-affinity IgG antibodies that compete with IgM antibodies (23), contributing to the lower agreement rates with IgM results. In agreement with previous research on the use of lateral flow assays in unvaccinated individuals (38, 39), we also report lower evaluator concordance when reading IgM compared with IgG bands. Jones and colleagues found that IgM bands are often weak positives (39), consistent with our data in vaccinated individuals, potentially resulting in an increased likelihood of a miscall and contributing to the low reader agreement.…”

Unexplained longitudinal variability in COVID-19 antibody status by Lateral Flow Immuno-Antibody testing

“…There is known to be substantial heterogeneity of COVID-19 antibody test results depending on the type of antibody probed and the kit used. (Kontou, Braliou et al 2020, Favresse, Eucher et al 2021, Jones, Mulchandani et al 2021). For some LFIAs, heterogeneity has additionally been shown between batches of testing kits, lab versus point of care use, and the use of venous versus capillary blood.…”