2022
DOI: 10.48550/arxiv.2202.02833
|View full text |Cite
Preprint
|
Sign up to set email alerts
|

CheXstray: Real-time Multi-Modal Data Concordance for Drift Detection in Medical Imaging AI

Abstract: Background/Motivation: Rapidly expanding Clinical AI applications worldwide have the potential to impact to all areas of medical practice. Medical imaging applications constitute a vast majority of approved clinical AI applications with more than 150 to date. Though healthcare systems are eager to adopt AI solutions after initial successes in model validation and deployment for clinical workflows, fundamental questions remain: Is the model still working as expected?, What is causing the change?, Is it time to … Show more

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...
1
1
1
1

Citation Types

0
4
0

Year Published

2022
2022
2024
2024

Publication Types

Select...
3
1

Relationship

0
4

Authors

Journals

citations
Cited by 4 publications
(4 citation statements)
references
References 20 publications
0
4
0
Order By: Relevance
“…To gain earlier insights into AI's accuracy, additional tools for assessing expected AI outcomes based on input data (e.g., determining whether the input data falls within or outside the training data distribution) or comparing the results of one AI model to those of other AI models simultaneously can be employed. 88 Autonomous AI should be designed to initiate actions that are transparent, identifiable, and discoverable. The capacity to disable the AI system swiftly and effectively in the event of failure is crucial for patient safety.…”
Section: Autonomous Aimentioning
confidence: 99%
“…To gain earlier insights into AI's accuracy, additional tools for assessing expected AI outcomes based on input data (e.g., determining whether the input data falls within or outside the training data distribution) or comparing the results of one AI model to those of other AI models simultaneously can be employed. 88 Autonomous AI should be designed to initiate actions that are transparent, identifiable, and discoverable. The capacity to disable the AI system swiftly and effectively in the event of failure is crucial for patient safety.…”
Section: Autonomous Aimentioning
confidence: 99%
“…Data shift is, of course, present in the data input to the AI model, and one possibility is to detect data shifts at the input stage. A recent radiology example is the data drift monitoring method for chest X-ray data by Soin et al [33], taking multi-modal input data into account.…”
Section: Input and Output Data Shift Detectionmentioning
confidence: 99%
“…Another aspect to consider is the risk that the identified potential issue is a false alarm or has negligible effect. For example, this is an inherent difficulty in data shift detection [33]. In general, the more one deploys automated quality issue indicators, the more you need to deal with in terms of drill-down efforts.…”
Section: Issue-targeted Scrutinymentioning
confidence: 99%
“…PyHealth [25] is a Python library that offers preprocessing across various EHR datasets and modelling for multiple clinical prediction tasks. Moreover, these tools fail to offer capabilities to evaluate model robustness to data shifts and existing tools for monitoring of clinical ML models like CheXstray [18] focus on specific domains and offer limited functionality.…”
Section: Related Workmentioning
confidence: 99%