Reliability of using single specialist annotation for designing and evaluating automatic segmentation methods: A skull stripping case study

Souza, Roberto; Lucena, Oeslle; Bento, Mariana; Garrafa, Julia; Appenzeller, Simone; Rittner, Letícia; Lotufo, Roberto de Alencar; Frayne, Richard

doi:10.1109/isbi.2018.8363820

Cited by 5 publications

(5 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Manual labeling, however, can be time-consuming, expensive, prone to errors, and may present with unacceptable intra- and inter-expert variation (Obuchowicz et al, 2020 ). There are different approaches to overcome these challenges, such as using consensus methods, like simultaneous truth and performance level estimation (STAPLE, Warfield et al, 2004 ), and unsupervised approaches (Shen, 2013 ; Souza et al, 2018 ).…”

Section: Data Accessmentioning

confidence: 99%

Deep Learning in Large and Multi-Site Structural Brain MR Imaging Datasets

Bento

Fantini

Park

et al. 2022

Front. Neuroinform.

Self Cite

View full text Add to dashboard Cite

Large, multi-site, heterogeneous brain imaging datasets are increasingly required for the training, validation, and testing of advanced deep learning (DL)-based automated tools, including structural magnetic resonance (MR) image-based diagnostic and treatment monitoring approaches. When assembling a number of smaller datasets to form a larger dataset, understanding the underlying variability between different acquisition and processing protocols across the aggregated dataset (termed “batch effects”) is critical. The presence of variation in the training dataset is important as it more closely reflects the true underlying data distribution and, thus, may enhance the overall generalizability of the tool. However, the impact of batch effects must be carefully evaluated in order to avoid undesirable effects that, for example, may reduce performance measures. Batch effects can result from many sources, including differences in acquisition equipment, imaging technique and parameters, as well as applied processing methodologies. Their impact, both beneficial and adversarial, must be considered when developing tools to ensure that their outputs are related to the proposed clinical or research question (i.e., actual disease-related or pathological changes) and are not simply due to the peculiarities of underlying batch effects in the aggregated dataset. We reviewed applications of DL in structural brain MR imaging that aggregated images from neuroimaging datasets, typically acquired at multiple sites. We examined datasets containing both healthy control participants and patients that were acquired using varying acquisition protocols. First, we discussed issues around Data Access and enumerated the key characteristics of some commonly used publicly available brain datasets. Then we reviewed methods for correcting batch effects by exploring the two main classes of approaches: Data Harmonization that uses data standardization, quality control protocols or other similar algorithms and procedures to explicitly understand and minimize unwanted batch effects; and Domain Adaptation that develops DL tools that implicitly handle the batch effects by using approaches to achieve reliable and robust results. In this narrative review, we highlighted the advantages and disadvantages of both classes of DL approaches, and described key challenges to be addressed in future studies.

show abstract

Section: Data Accessmentioning

confidence: 99%

Deep Learning in Large and Multi-Site Structural Brain MR Imaging Datasets

Bento

Fantini

Park

et al. 2022

Front. Neuroinform.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Although not a hippocampus segmentation work, (Lucena et al, 2018) inspired our consensus strategy that involves the use of multiple FCNNs performing segmentation over different MRI orientations, merged into a single final volume. While his method uses another network to produce the final consensus, in our post processing we simply add the activation heatmap of each FCNN, apply a pre-defined threshold, and perform 3D labeling to eliminate all but the two bigger connected components, which is shown to improve performance significantly.…”

Section: Related Workmentioning

confidence: 99%

“…Standard procedure in this cases is to perform a MRI scan of the brain and have experts analyze the shape of the hippocampus. To this day, manual segmentation is still a gold standard, even though interrater variability is a concerning problem (Souza et al, 2018a). However, manual segmentation still takes time and needs to be performed by specialized personnel.…”

Section: Introductionmentioning

confidence: 99%

Extended 2D Consensus Hippocampus Segmentation

Carmo,

Silva,

Yasuda

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Hippocampus segmentation plays a key role in diagnosing various brain disorders such as Alzheimer's disease, epilepsy, multiple sclerosis, cancer, depression and others. Nowadays, segmentation is still mainly performed manually by specialists. Segmentation done by experts is considered to be a gold-standard when evaluating automated methods, buts it is a time consuming and arduos task, requiring specialized personnel. In recent years, efforts have been made to achieve reliable automated segmentation. For years the best performing authomatic methods were multi atlas based with around 80-85% Dice coefficient and very time consuming, but machine learning methods are recently rising with promising time and accuracy performance. A method for volumetric hippocampus segmentation is presented, based on the consensus of tri-planar U-Net inspired fully convolutional networks (FCNNs), with some modifications, including residual connections, VGG weight transfers, batch normalization and a patch extraction technique employing data from neighbor patches. A study on the impact of our modifications to the classical U-Net architecture was performed. Our method achieves cutting edge performance in our dataset, with around 96% volumetric Dice accuracy in our test data. In a public validation dataset, HARP, we achieve 87.48% DICE. GPU execution time is in the order of seconds per volume, and source code is publicly available. Also, masks are shown to be similar to other recent state-of-the-art hippocampus segmentation methods in a third dataset, without manual annotations.

show abstract

“…Also, manual annotation is a subjective procedure. For instance, there are more than a dozen protocols for hippocampus segmentation and different protocols provide up to 2.5fold volume difference (SOUZA et al, 2018). As a consequence, intra/inter rater variability affect the validation results in such data driven methods.…”

Section: Motivationmentioning

confidence: 99%

Deep learning for brain analysis in MR imaging

Lucena¹

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) are one branch of deep learning that have performed successfully in many brain magnetic resonance (MR) imaging analysis. CNNs are representation-learning methods with stacked layers comprised of a convolution operation followed by a non-linear activation and pooling layers. In these networks, each layer outputs a higher and more abstract representation from a given input, in which the weights of the convolutional layers are learned by an optimization problem. In this work, we tackled two problems using deep-learning-based approaches: skull-stripping (SS) and tractography. We Ąrstly proposed a full CNN-based SS trained with what we refer to as silver standard masks. Segmenting brain tissue from non-brain tissue is a process known as brain extraction or skull-stripping. Silver standard masks are generated by forming the consensus from a set of eight, public, non-deep-learning-based SS methods using the algorithm Simultaneous Truth and Performance Level Estimation (STAPLE). Our approach reached state-of-the-art performance, generalized optimally, decreased inter-/intra-rater variability, and avoided CNN segmentation overĄtting towards one speciĄc manual annotation. Secondly, we investigated a CNN-based tractography solution for epilepsy surgery. The main goal of this analysis was to structure a baseline for a deep-learning-basedregression to predict white matter Ąber orientations. Tractography is a visualization of the white matter Ąbers or tracts; its goal in presurgical planing is simply to identify the position of eloquent pathways, such as the motor, sensory, and language tracts to reduce the risk of damaging these critical structures. We performed analysis cross-validation using only in a single patient per time, and also, training with data from 10 patients for training the CNN. Our results were not optimal, however, the tracts tended to be of a similar length and converged to the mean Ąber tract locations. Additionally, to the best of our knowledge, our method is the Ąrst approach that investigates CNNs for tractography, and thus, our work is a baseline for this topic.

show abstract

Reliability of using single specialist annotation for designing and evaluating automatic segmentation methods: A skull stripping case study

Cited by 5 publications

References 17 publications

Deep Learning in Large and Multi-Site Structural Brain MR Imaging Datasets

Deep Learning in Large and Multi-Site Structural Brain MR Imaging Datasets

Extended 2D Consensus Hippocampus Segmentation

Deep learning for brain analysis in MR imaging

Contact Info

Product

Resources

About