Toward a Common Performance and Effectiveness Terminology for Digital Proximity Tracing Applications

Abstract

Digital proximity tracing (DPT) for Sars-CoV-2 pandemic mitigation is a complex intervention with the primary goal to notify app users about possible risk exposures to infected persons. DPT not only relies on the technical functioning of the proximity tracing application and its backend server, but also on seamless integration of health system processes such as laboratory testing, communication of results (and their validation), generation of notification codes, manual contact tracing, and management of app-notified users. Policymakers and DPT operators need to know whether their system works as expected in terms of speed or yield (performance) and whether DPT is making an effective contribution to pandemic mitigation (also in comparison to and beyond established mitigation measures, particularly manual contact tracing). Thereby, performance and effectiveness are not to be confused. Not only are there conceptual differences but also diverse data requirements. For example, comparative effectiveness measures may require information generated outside the DPT system, e.g., from manual contact tracing. This article describes differences between performance and effectiveness measures and attempts to develop a terminology and classification system for DPT evaluation. We discuss key aspects for critical assessments of whether the integration of additional data measurements into DPT apps may facilitate understanding of performance and effectiveness of planned and deployed DPT apps. Therefore, the terminology and a classification system may offer some guidance to DPT system operators regarding which measurements to prioritize. DPT developers and operators may also make conscious decisions to integrate measures for epidemic monitoring but should be aware that this introduces a secondary purpose to DPT. Ultimately, the integration of further information (e.g., regarding exact exposure time) into DPT involves a trade-off between data granularity and linkage on the one hand, and privacy on the other. More data may lead to better epidemiological information but may also increase the privacy risks associated with the system, and thus decrease public DPT acceptance. Decision-makers should be aware of the trade-off and take it into account when planning and developing DPT systems or intending to assess the added value of DPT relative to the existing contact tracing systems.

Using Venn Diagrams to Evaluate Digital Contact Tracing: Panel Survey Analysis

“…This approach makes use of Venn diagrams to visualize the co-occurrence of SARS-CoV-2 outcomes of interest based on digital proximity tracing app use [ 21 ]. To construct the Venn diagrams, as recently formalized [ 21 ], requirements are established to define the appropriate data sets and time points, as well as identify subpopulations, to calculate digital proximity tracing app effectiveness.…”

“…This approach makes use of Venn diagrams to visualize the co-occurrence of SARS-CoV-2 outcomes of interest based on digital proximity tracing app use [ 21 ]. To construct the Venn diagrams, as recently formalized [ 21 ], requirements are established to define the appropriate data sets and time points, as well as identify subpopulations, to calculate digital proximity tracing app effectiveness. Based on our experience and extensive reporting of key indicators to mitigate the spread of SARS-CoV-2 [ 2 , 19 , 20 ], we proposed 4 attributes for Venn diagram development to facilitate the identification of subpopulations of interest: (1) having been tested for SARS-CoV-2, (2) having a positive SARS-CoV-2 test result, (3) having received an exposure notification, and (4) having entered into quarantine ( Table 1 ).…”

Performance of the Swiss Digital Contact-Tracing App Over Various SARS-CoV-2 Pandemic Waves: Repeated Cross-sectional Analyses

“…Second, the privacy-preserving architecture of digital proximity-tracing apps, particularly those that follow the Decentralized Privacy-Preserving Proximity Tracing (DP-3T) blueprint [ 10 ], provides only limited, nonidentifiable data for conducting effectiveness analyses. Lastly, additional relevant data generated, for example, through manual contact tracing, information hotlines, and testing centers, henceforth described as “points of contact for app users,” are often dispersed across different systems and not readily available due to privacy regulations [ 11 ].…”

Digital Contact Tracing Apps for COVID-19: Development of a Citizen-Centered Evaluation Framework

“…The Effectiveness pillar refers to the degree to which the app is successful in its core aims: accurately detecting close contacts and thus providing “notification to other app users with potential exposure risks to an infected app user” [ 98 ]. It contains three high-level attributes (see Table 2 ), the first of which ( Effective Reporting ) refers to concerns related to accurate detection, and the second of which ( Effective Results ) refers to providing notification to other app users with potential exposure risks, a concern provisionally referred to as “performance” by other commentators in the field [ 77 ]. The third attribute ( Effective Engagement ) refers to the “other app users” and “infected app users” in the definition, specifically focusing on the level of app adoption by citizens.…”

“…Concerns include detecting and sharing close contacts, providing relevant information to citizens, and assessing their reactions to that information. This pillar was informed by drawing and expanding on the definition of effectiveness in CTAs, provided by Lueks et al [ 77 ], and by considering Vokinger et al’s [ 58 ] framework, which also explicitly tackles this concern.…”