End-to-End diagnosis of breast biopsy images with transformers

Mehta, Sachin; Lu, Ximing; Wu, Wenjun; Weaver, Donald L.; Hajishirzi, Hannaneh; Elmore, Joann G.; Shapiro, Linda G.

doi:10.1016/j.media.2022.102466

Cited by 20 publications

(8 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figure 9 shows some examples of SOTA transformer architectures developed for histopathological image classification. For breast cancer histopathological image classification, DCET-Net [ 72 ] proposed a dual-stream convolution-expanded transformer architecture; Breast-Net [ 51 ] explores the ability of ensemble learning techniques using four Swin transformer architectures; HATNet [ 52 ] uses end-to-end vision transformers with a self-attention mechanism; ScoreNet [ 16 ] developed an efficient transformer-based architecture that integrates a coarse-grained global attention framework with a fine-grained local attention mechanism framework; LGVIT [ 73 ] built a local–global ViT model by introducing a new local–global MHSA mechanism and a ghost geed-forward network block into the network; dMIL-transformer [ 53 ] developed a two-stage double max–min multiple-instance learning (MIL) transformer architecture that combines both the spatial and morphological information of the cancer regions. Other than breast cancer classification, transformers have also been applied to other histopathological image cancer classification tasks, such as bone cancer classification (NRCA-FCFL [ 74 ]), brain cancer classification (ViT-WSI [ 17 ], ASI-DBNet [ 54 ], Ding et al [ 55 ]), colorectal cancer classification (MIST [ 75 ], DT-DSMIL [ 56 ]), gastric cancer classification (IMGL-VTNet [ 57 ]), kidney subtype classification (i-ViT [ 59 ], tRNAsformer [ 58 ]), thymoma or thymic carcinoma classification (MC-ViT [ 76 ]), lung cancer classification (GTP [ 46 ], FDTrans [ 60 ]), skin cancer classification (Wang et al [ 45 ]), and thyroid cancer classification (Wang et al [ 77 ], PyT2T-ViT [ 41 ], Wang et al [ 78 ]) using different transformer-based architectures.…”

Section: Current Progressmentioning

confidence: 99%

A survey of Transformer applications for histopathological image analysis: New developments and future directions

Atabansi,

Nie,

Liu

et al. 2023

BioMed Eng OnLine

View full text Add to dashboard Cite

Transformers have been widely used in many computer vision challenges and have shown the capability of producing better results than convolutional neural networks (CNNs). Taking advantage of capturing long-range contextual information and learning more complex relations in the image data, Transformers have been used and applied to histopathological image processing tasks. In this survey, we make an effort to present a thorough analysis of the uses of Transformers in histopathological image analysis, covering several topics, from the newly built Transformer models to unresolved challenges. To be more precise, we first begin by outlining the fundamental principles of the attention mechanism included in Transformer models and other key frameworks. Second, we analyze Transformer-based applications in the histopathological imaging domain and provide a thorough evaluation of more than 100 research publications across different downstream tasks to cover the most recent innovations, including survival analysis and prediction, segmentation, classification, detection, and representation. Within this survey work, we also compare the performance of CNN-based techniques to Transformers based on recently published papers, highlight major challenges, and provide interesting future research directions. Despite the outstanding performance of the Transformer-based architectures in a number of papers reviewed in this survey, we anticipate that further improvements and exploration of Transformers in the histopathological imaging domain are still required in the future. We hope that this survey paper will give readers in this field of study a thorough understanding of Transformer-based techniques in histopathological image analysis, and an up-to-date paper list summary will be provided at https://github.com/S-domain/Survey-Paper.

show abstract

Section: Current Progressmentioning

confidence: 99%

A survey of Transformer applications for histopathological image analysis: New developments and future directions

Atabansi,

Nie,

Liu

et al. 2023

BioMed Eng OnLine

View full text Add to dashboard Cite

show abstract

“…For a reliable diagnostic system, it is important to obtain representations that reflect both the content and context of the input biopsy image. HATNet, a system we developed originally for breast biopsy analysis, achieved this using a top-down and bottom-up approach (Mehta et al, 2022 ). Pathologists describe using a different viewing behavior before making their diagnosis of breast tissue compared with their assessment of skin biopsy images.…”

Section: Diagnosismentioning

confidence: 99%

Automated analysis of whole slide digital skin biopsy images

Nofallah

Wu²,

Liu³

et al. 2022

Front. Artif. Intell.

Self Cite

View full text Add to dashboard Cite

A rapidly increasing rate of melanoma diagnosis has been noted over the past three decades, and nearly 1 in 4 skin biopsies are diagnosed as melanocytic lesions. The gold standard for diagnosis of melanoma is the histopathological examination by a pathologist to analyze biopsy material at both the cellular and structural levels. A pathologist's diagnosis is often subjective and prone to variability, while deep learning image analysis methods may improve and complement current diagnostic and prognostic capabilities. Mitoses are important entities when reviewing skin biopsy cases as their presence carries prognostic information; thus, their precise detection is an important factor for clinical care. In addition, semantic segmentation of clinically important structures in skin biopsies might help the diagnosis pipeline with an accurate classification. We aim to provide prognostic and diagnostic information on skin biopsy images, including the detection of cellular level entities, segmentation of clinically important tissue structures, and other important factors toward the accurate diagnosis of skin biopsy images. This paper is an overview of our work on analysis of digital whole slide skin biopsy images, including mitotic figure (mitosis) detection, semantic segmentation, diagnosis, and analysis of pathologists' viewing patterns, and with new work on melanocyte detection. Deep learning has been applied to our methods for all the detection, segmentation, and diagnosis work. In our studies, deep learning is proven superior to prior approaches to skin biopsy analysis. Our work on analysis of pathologists' viewing patterns is the only such work in the skin biopsy literature. Our work covers the whole spectrum from low-level entities through diagnosis and understanding what pathologists do in performing their diagnoses.

show abstract

“…However, this approach overlooks the histological and morphological features of tumor cells with its matrix. With the development of digital pathological equipment and decreasing storage costs, the application of digital pathology has generated considerable enthusiasm as a useful tool to assist pathologists in reporting the pathological results of tumor tissue . Importantly, digital H-E images combined with machine learning methods could objectively and automatically extract quantitative pathological information, so-called pathomics .…”

Section: Introductionmentioning

confidence: 99%

“…With the development of digital pathological equipment and decreasing storage costs, the application of digital pathology has generated considerable enthusiasm as a useful tool to assist pathologists in reporting the pathological results of tumor tissue. 8,9 Importantly, digital H-E images combined with machine learning methods could objectively and automatically extract quantitative pathological information, so-called pathomics. 10,11 Previous studies have indicated that pathomics combined with artificial intelligence algorithms can be used to estimate the prognosis of hepatocellular cancer, renal cell cancer, and colorectal cancer lung metastasis.…”

mentioning

confidence: 99%

Pathomics Signature for Prognosis and Chemotherapy Benefits in Stage III Colon Cancer

Jiang,

Wang,

Dong

et al. 2024

JAMA Surg

View full text Add to dashboard Cite

ImportanceThe current TNM staging system may not provide adequate information for prognostic purposes and to assess the potential benefits of chemotherapy for patients with stage III colon cancer.ObjectiveTo develop and validate a pathomics signature to estimate prognosis and benefit from chemotherapy using hematoxylin-eosin (H-E)–stained slides.Design, Setting, and ParticipantsThis retrospective prognostic study used data from consecutive patients with histologically confirmed stage III colon cancer at 2 medical centers between January 2012 and December 2015. A total of 114 pathomics features were extracted from digital H-E–stained images from Nanfang Hospital of Southern Medical University, Guangzhou, China, and a pathomics signature was constructed using a least absolute shrinkage and selection operator Cox regression model in the training cohort. The associations of the pathomics signature with disease-free survival (DFS) and overall survival (OS) were evaluated. Patients at the Sixth Affiliated Hospital, Sun Yat-sen University, Guangzhou, China, formed the validation cohort. Data analysis was conducted from September 2022 to March 2023.Main Outcomes and MeasuresThe prognostic accuracy of the pathomics signature as well as its association with chemotherapy response were evaluated.ResultsThis study included 785 patients (mean [SD] age, 62.7 [11.1] years; 437 [55.7%] male). A pathomics signature was constructed based on 4 features. Multivariable analysis revealed that the pathomics signature was an independent factor associated with DFS (hazard ratio [HR], 2.46 [95% CI, 2.89-4.13]; P &lt; .001) and OS (HR, 2.78 [95% CI, 2.34-3.31]; P &lt; .001) in the training cohort. Incorporating the pathomics signature into pathomics nomograms resulted in better performance for the estimation of prognosis than the traditional model in a concordance index comparison in the training cohort (DFS: HR, 0.88 [95% CI, 0.86-0.89] vs HR, 0.73 [95% CI, 0.71-0.75]; P &lt; .001; OS: HR, 0.85 [95% CI, 0.84-0.86] vs HR, 0.74 [95% CI, 0.72-0.76]; P &lt; .001) and validation cohort (DFS: HR, 0.83 [95% CI, 0.82-0.85] vs HR, 0.70 [95% CI, 0.67-0.72]; P &lt; .001; OS: HR, 0.80 [95% CI, 0.78-0.82] vs HR, 0.69 [0.67-0.72]; P &lt; .001). Further analysis revealed that patients with a low pathomics signature were more likely to benefit from chemotherapy (eg, combined cohort: DFS: HR, 0.44 [95% CI, 0.28-0.69]; P = .001; OS: HR, 0.43 [95% CI, 0.29-0.64]; P &lt; .001).Conclusions and RelevanceThese findings suggest that a pathomics signature could help identify patients most likely to benefit from chemotherapy in stage III colon cancer.

show abstract

End-to-End diagnosis of breast biopsy images with transformers

Cited by 20 publications

References 47 publications

A survey of Transformer applications for histopathological image analysis: New developments and future directions

A survey of Transformer applications for histopathological image analysis: New developments and future directions

Automated analysis of whole slide digital skin biopsy images

Pathomics Signature for Prognosis and Chemotherapy Benefits in Stage III Colon Cancer

Contact Info

Product

Resources

About