Evaluation of Diagnostic Accuracy of the System for Pulmonary Tuberculosis Screening Based on Artificial Neural Networks

Морозов, С. П.; VLАDZIMIRSKIY, А. V.; Ледихова, Н. В.; Sokolina, Irina; KULBERG, N. S; Gombolevskiy, Victor A.

doi:10.21292/2075-1230-2018-96-8-42-49

Cited by 9 publications

(4 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In the prospective study, AUC decreased to 0.78, followed by specificity (prior – 0.925, retrospectively – 0.87, prospectively – 0.68) 23 . We also previously tested the FBM (FLG) service on two datasets with a balance of normal-to-pathology 50/50 and 95/5 with resulting AUC of 0.74 and 0.64, sensitivity – 0.872 and 0.75, specificity – 0.60 and 0.535, concluding that the accuracy varied depending on the class balance 24 . At the first stage of the Moscow Experiment, the AUC was 0.9 – higher than the prior study.…”

Section: Discussionmentioning

confidence: 99%

Diagnostic Accuracy of Artificial Intelligence for Analysis of 1.3 Million Medical Imaging Studies: The Moscow Experiment on Computer Vision Technologies

Morozov,

Vladzymyrskyy,

Ledikhova

et al. 2023

Preprint

View full text Add to dashboard Cite

Objective: to assess the diagnostic accuracy of services based on computer vision technologies at the integration and operation stages in Moscow's Unified Radiological Information Service (URIS). Methods: this is a multicenter diagnostic study of artificial intelligence (AI) services with retrospective and prospective stages. The minimum acceptable criteria levels for the index test were established, justifying the intended clinical application of the investigated index test. The Experiment was based on the infrastructure of the URIS and United Medical Information and Analytical System (UMIAS) of Moscow. Basic functional and diagnostic requirements for the artificial intelligence services and methods for monitoring technological and diagnostic quality were developed. Diagnostic accuracy metrics were calculated and compared. Results: based on the results of the retrospective study, we can conclude that AI services have good result reproducibility on local test sets. The highest and at the same time most balanced metrics were obtained for AI services processing CT scans. All AI services demonstrated a pronounced decrease in diagnostic accuracy in the prospective study. The results indicated a need for further refinement of AI services with additional training on the Moscow population datasets. Conclusions: the diagnostic accuracy and reproducibility of AI services on the reference data are sufficient, however, they are insufficient on the data in routine clinical practice. The AI services that participated in the experiment require a technological improvement, additional training on Moscow population datasets, technical and clinical trials to get a status of a medical device.

show abstract

Section: Discussionmentioning

confidence: 99%

Diagnostic Accuracy of Artificial Intelligence for Analysis of 1.3 Million Medical Imaging Studies: The Moscow Experiment on Computer Vision Technologies

Morozov,

Vladzymyrskyy,

Ledikhova

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…При сравнении результатов нашего исследования с данными других работ отмечено, что показатели чувствительности, специфичности, отношения правдоподобия положительного результата, отношения правдоподобия отрицательного результата, прогностической ценности положительного результата, прогностической ценности отрицательного результата находятся в одном диапазоне, но колеблются в пределах 5-10%, что может быть обусловлено как качеством обучения программных продуктов, так и разностью выборок, примененных для тестирования. Это требует дальнейшего исследования на большем количестве выборок [6,8,14].…”

Section: Discussionunclassified

“…Эти факторы объясняют большой интерес разработчиков программного обеспечения к созданию диагностических алгоритмов, позволяющих проводить автоматический анализ рентгенологических изображений легких. Применение таких алгоритмов предполагает более эффективное обнаружение патологических изменений [6][7][8].…”

unclassified

Evaluation of diagnostic accuracy of the automatic system for the analysis of digital lung X-ray for detection of spherical masses

Gavrilov

Smolnikova

2021

Alʹm. klin. med.

View full text Add to dashboard Cite

Rationale: Most data on the effectiveness of systems for the analysis of digital X-ray images have been provided by their developers and require a high-quality validation in databases prepared independently of the developer.Aim: To analyze the information content of automatic identification of spherical lung masses with digital X-ray imaging using one of the widely available diagnostic algorithms on publicly unaccessible reference datasets.Materials and methods: The study was based on the recognition and analysis of digital X-ray images from two publicly inaccessible reference datasets that have the state registration (Russian Federation) with one of the publicly available diagnostic algorithms (FutureMed Analyzer). The study was performed using two models of X-ray screening as examples: Model 1 consisted of 100 X-ray images of the lungs with a normal: abnormal ratio of 94%: 6%; Model 2 consisted of 5150 chest X-ray images with a normal: abnormal ratio of 97%: 3%.Results: According to the results of the analysis of the X-ray images with the diagnostic system, 98% of the images were correctly interpreted with Model 1 and 95% of the images, with Model 2. 83% of the cases from Model 1 and 69% from Model 2% were interpreted as images with lung abnormalities. The percentage of correct answers for differentiation of the chest X-ray images into two categories (normal vs. abnormal) for Model 1 and Model 2 was 95% and 98%, respectively. The sensitivity for detection of abnormal masses ranged from 69% to 83%. The specificity was 99% for the Model 1 chest X-ray images and 96% for the Model 2 chest X-ray images. The underdiagnosis rate was quite low ranging for Model 1 – 17%, and for Model 2 – 31%. The area under the curve for Model 1 was 0.91 and for Model 2 0.85.Conclusion: The diagnostic efficiency of the automatic image analysis based on the convolutional neuronal networks approaches that of the radiologists. This system of automatic identification of abnormalities was unable to solve the most complex problems of detecting low density spherical masses (like "ground glass" area on computed tomography) and that of shadow summation for abnormalities located in such difficult to interpret zones as lung apices, clavicles, ribs, etc. To select a suitable system, medical institutions need to conduct preliminary testing in their own models equivalent to the studies performed in a given institution (parameters for radiography, nature and frequency of abnormalities).

show abstract

“…Однако в литературе в настоящее время большинство данных об эффективности этих систем предоставленысамими разработчиками, существуют лишь единичные публикации, посвященные независимой оценке таких систем [5].…”

Section: Research Funding and Conflict Of Interestunclassified

Diagnostic Efficiency of Various Systems for Automatic Analysis of Radiographs in the Detection of Lung Nodule

Smolnikova

Gavrilov

Yаblonskiy

2022

jour

View full text Add to dashboard Cite

The purpose of the study was to compare the effectiveness of various artificial intelligence systems for detecting foci and rounded lesions in the lungs. For testing, we selected four software products based on convolutional neural networks, positioning themselves as a sensitive system for evaluating digital chest radiographs. An analytical validation method was used for clinical evaluation. For diagnostics, 3 data samples were formed with the identification of signs of diseases (sample 1–5150 radiographs, detection of pathological changes 3 %; sample 2–100 radiographs, detection of pathological changes 6 %; sample 3–300 radiographs, detection of the prevalence of pathological changes 50 %). None of the software products passed the AUC threshold of 0.811 on all three samples. In all three samples, all software products have high accuracy and high sensitivity in detecting round formations, which leads to rare cases of overdiagnosis and special cases of underdiagnosis. The use of digital X-ray image analysis systems based on artificial intelligence technologies is a promising direction for high-quality diagnostics, primarily when considering their young radiologists as an additional opinion.

show abstract

Evaluation of Diagnostic Accuracy of the System for Pulmonary Tuberculosis Screening Based on Artificial Neural Networks

Cited by 9 publications

References 20 publications

Diagnostic Accuracy of Artificial Intelligence for Analysis of 1.3 Million Medical Imaging Studies: The Moscow Experiment on Computer Vision Technologies

Diagnostic Accuracy of Artificial Intelligence for Analysis of 1.3 Million Medical Imaging Studies: The Moscow Experiment on Computer Vision Technologies

Evaluation of diagnostic accuracy of the automatic system for the analysis of digital lung X-ray for detection of spherical masses

Diagnostic Efficiency of Various Systems for Automatic Analysis of Radiographs in the Detection of Lung Nodule

Contact Info

Product

Resources

About