An important need exists for strategies to perform rigorous objective clinical-task-based evaluation of artificial intelligence (AI) algorithms for nuclear medicine. To address this need, we propose a 4-class framework to evaluate AI algorithms for promise, technical task-specific efficacy, clinical decision making, and postdeployment efficacy. We provide best practices to evaluate AI algorithms for each of these classes. Each class of evaluation yields a claim that provides a descriptive performance of the AI algorithm. Key best practices are tabulated as the RELAINCE (Recommendations for EvaLuation of AI for NuClear medicinE) guidelines. The report was prepared by the Society of Nuclear Medicine and Molecular Imaging AI Task Force Evaluation team, which consisted of nuclear-medicine physicians, physicists, computational imaging scientists, and representatives from industry and regulatory agencies.

Objectives N95 filtering facepiece respirators (N95 FFRs) and surgical masks are comprised of multiple layers of nonwoven polypropylene. Tight-fitting N95 FFRs are respiratory protective devices (RPDs) designed to efficiently filter aerosols. During the COVID-19 pandemic, health care workers (HCWs) throughout the world continue to face shortages of disposable N95 FFRs. Existing version of widely available FDA cleared loose-fitting surgical masks with straps do not provide reliable protection against aerosols. We tested the faceseal of a modified strapless form-fitting sealed version of surgical mask using quantitative fit testing (QNFT) and compared the performance of this mask with that of N95 FFRs and unmodified loose-fitting surgical masks. Methods Twenty HCWs participated in the study (10 women; 10 men; age 23–59 years). To create the sealed surgical masks, we removed the straps from loose-fitting surgical masks, made new folds, and used adhesive medical tape to secure the new design. All participants underwent QNFT with a loose-fitting surgical mask, the sealed surgical mask, and an N95 FFR; fit factors were recorded. Each QNFT was performed using a protocol of four exercises: (i) bending over, (ii) talking, (iii) moving head side to side, and (iv) moving head up and down. When the overall fit factor for the sealed surgical mask or N95 FFR was <100, the participant retook the test. Participants scored the breathability and comfort of the sealed surgical mask and N95 FFR on a visual analog scale (VAS) ranging from 0 (unfavorable) to 10 (favorable). Results The median fit factor for the sealed surgical mask (53.8) was significantly higher than that of the loose-fitting surgical mask (3.0) but lower than that of the N95 FFR (177.0) (P < 0.001), equating to significantly lower inward leakage of ambient aerosols (measuring 0.04–0.06 µm) with the sealed surgical mask (geometric mean 1.79%; geometric standard deviation 1.45%; range 0.97–4.03%) than with the loose-fitting surgical mask (29.5%; 2.01%; 25–100.0%) but still higher than with the N95 FFR (0.66%; 1.46%; 0.50–1.97%) (P < 0.001). Sealed surgical masks led to a marked reduction (range 60–98%) in inward leakage of aerosols in all the participants, compared to loose-fitting surgical masks. Among the exercises, talking had a greater effect on reducing overall fit factor for the sealed surgical mask than for the N95 FFR; when talking was excluded, the fit factor for the sealed surgical mask improved significantly (median 53.8 to 81.5; P < 0.001). The sealed surgical mask, when compared with the N95 FFR, offered better reported breathability (median VAS 9 versus 5; P < 0.001) and comfort (9 versus 5; P < 0.001). Conclusions Widely available loose-fitting surgical masks can be easily modified to achieve faceseal with adhesives. Unlike loose-fitting surgical masks, sealed surgical masks can markedly reduce inward leakage of aerosols and may therefore offer useful levels of respiratory protection during an extreme shortage of N95 FFRs and could benefit HCWs who cannot comply with N95 FFRs due to intolerance. However, because a wide range of surgical masks is commercially available, individual evaluation of such masks is highly recommended before sealed versions are used as RPDs.

No abstract