Magician’s Corner: 9. Performance Metrics for Machine Learning                     Models

Erickson, Bradley J.; Kitamura, Felipe

doi:10.1148/ryai.2021200126

Cited by 137 publications

(66 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The variables to be predicted (outputs) were also be defined, which may be intermediate, such as organ segmentation, or a final clinical event related to the clinical question. Indicates the metrics used to evaluate the goodness of fit of the trained model on the test data set, such as precision, accuracy or specificity [ 25 ]. Describes all the training, hyperparameter tuning and testing procedures of the AI solution in sufficient detail so that it can be replicated by other researchers in other environments.…”

Section: Resultsmentioning

confidence: 99%

MAIC–10 brief quality checklist for publications using artificial intelligence and medical images

et al. 2023

View full text Add to dashboard Cite

The use of artificial intelligence (AI) with medical images to solve clinical problems is becoming increasingly common, and the development of new AI solutions is leading to more studies and publications using this computational technology. As a novel research area, the use of common standards to aid AI developers and reviewers as quality control criteria will improve the peer review process. Although some guidelines do exist, their heterogeneity and extension advocate that more explicit and simple schemes should be applied on the publication practice. Based on a review of existing AI guidelines, a proposal which collects, unifies, and simplifies the most relevant criteria was developed. The MAIC-10 (Must AI Criteria-10) checklist with 10 items was implemented as a guide to design studies and evaluate publications related to AI in the field of medical imaging. Articles published in Insights into Imaging in 2021 were selected to calculate their corresponding MAIC-10 quality score. The mean score was found to be 5.6 ± 1.6, with critical items present in most articles, such as “Clinical need”, “Data annotation”, “Robustness”, and “Transparency” present in more than 80% of papers, while improvements in other areas were identified. MAIC-10 was also observed to achieve the highest intra-observer reproducibility when compared to other existing checklists, with an overall reduction in terms of checklist length and complexity. In summary, MAIC-10 represents a short and simple quality assessment tool which is objective, robust and widely applicable to AI studies in medical imaging.

show abstract

Section: Resultsmentioning

confidence: 99%

MAIC–10 brief quality checklist for publications using artificial intelligence and medical images

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Herein, we evaluated the performance of our model in the testing set using the most common metrics for classification problems: F1 score, Area under the ROC curve, accuracy, precision and recall 22 . These metrics were calculated in a one-vs-rest manner .i.e.…”

Section: Methodsmentioning

confidence: 99%

Contrast phase recognition in liver computer tomography using deep learning

Rocha

Ferreira

Vianna³

et al. 2022

Sci Rep

View full text Add to dashboard Cite

Hepatocellular carcinoma (HCC) has become the 4th leading cause of cancer-related deaths, with high social, economical and health implications. Imaging techniques such as multiphase computed tomography (CT) have been successfully used for diagnosis of liver tumors such as HCC in a feasible and accurate way and its interpretation relies mainly on comparing the appearance of the lesions in the different contrast phases of the exam. Recently, some researchers have been dedicated to the development of tools based on machine learning (ML) algorithms, especially by deep learning techniques, to improve the diagnosis of liver lesions in imaging exams. However, the lack of standardization in the naming of the CT contrast phases in the DICOM metadata is a problem for real-life deployment of machine learning tools. Therefore, it is important to correctly identify the exam phase based only on the image and not on the exam metadata, which is unreliable. Motivated by this problem, we successfully created an annotation platform and implemented a convolutional neural network (CNN) to automatically identify the CT scan phases in the HCFMUSP database in the city of São Paulo, Brazil. We improved this algorithm with hyperparameter tuning and evaluated it with cross validation methods. Comparing its predictions with the radiologists annotation, it achieved an accuracy of 94.6%, 98% and 100% in the testing dataset for the slice, volume and exam evaluation, respectively.

show abstract

“… Specificity Specificity, also known as the true negative rate (TNR), measures the proportion of actual negatives that are correctly identified as such. (Erickson & Kitamura, 2021). It is the opposite of recall.…”

Section:  Precisionmentioning

confidence: 99%

Machine Learning Application in G.I.S. and Remote Sensing: An Overview

Upreti¹

2022

Preprint

View full text Add to dashboard Cite

Machine learning (ML) is a subdivision of artificial intelligence in which the machine learns from machine-readable data and information. It uses data, learns the pattern and predicts the new outcomes. Its popularity is growing because it helps to understand the trend and provides a solution that can be either a model or a product. Applications of ML algorithms have increased drastically in G.I.S. and remote sensing in recent years. It has a broad range of applications, from developing energy-based models to assessing soil liquefaction to creating a relation between air quality and mortality. Here, in this paper, we discuss the most popular supervised ML models (classification and regression) in G.I.S. and remote sensing. The motivation for writing this paper is that ML models produce higher accuracy than traditional parametric classifiers, especially for complex data with many predictor variables. This paper provides a general overview of some popular supervised non-parametric ML models that can be used in most of the G.I.S. and remote sensing-based projects. We discuss classification (Naïve Bayes (NB), Support Vector Machine (SVM), Random Forest (RF), Decision Trees (DT)) and regression models (Random Forest (RF), Support Vector Machine (SVM), Linear and Non-Linear) here. Therefore, the article can be a guide to those interested in using ML models in their G.I.S. and remote sensing-based projects

show abstract

Magician’s Corner: 9. Performance Metrics for Machine Learning Models

Cited by 137 publications

References 2 publications

MAIC–10 brief quality checklist for publications using artificial intelligence and medical images

MAIC–10 brief quality checklist for publications using artificial intelligence and medical images

Contrast phase recognition in liver computer tomography using deep learning

Machine Learning Application in G.I.S. and Remote Sensing: An Overview

Contact Info

Product

Resources

About