AW3M: An auto-weighting and recovery framework for breast cancer diagnosis using multi-modal ultrasound

Huang, Ruobing; Lin, Zhiping; Dou, Haoran; Wang, Jian; Miao, Jun; Zhou, Guangquan; Jia, Xiaohong; Xu, Wenwen; Mei, Zihan; Dong, Yijie; Zhou, Jianqiao; Ni, Dong

doi:10.1016/j.media.2021.102137

Cited by 32 publications

(12 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Gu et al 20 designed a DL model comprising multi‐fusion layers to obtain modal‐specific features and correlate information, respectively. Huang et al 21 developed a framework to utilize B‐mode, SWE‐mode, Doppler‐mode, and SE‐mode breast US images to assist breast cancer diagnosis. However, these methods require training the model for deep features and classifiers separately to achieve ensembled classification results.…”

Section: Related Workmentioning

confidence: 99%

Deep learning‐based multimodal fusion network for segmentation and classification of breast cancers using B‐mode and elastography ultrasound images

Misra

Yoon

Kim

et al. 2022

Bioengineering & Transla Med

View full text Add to dashboard Cite

Ultrasonography is one of the key medical imaging modalities for evaluating breast lesions. For differentiating benign from malignant lesions, computer‐aided diagnosis (CAD) systems have greatly assisted radiologists by automatically segmenting and identifying features of lesions. Here, we present deep learning (DL)‐based methods to segment the lesions and then classify benign from malignant, utilizing both B‐mode and strain elastography (SE‐mode) images. We propose a weighted multimodal U‐Net (W‐MM‐U‐Net) model for segmenting lesions where optimum weight is assigned on different imaging modalities using a weighted‐skip connection method to emphasize its importance. We design a multimodal fusion framework (MFF) on cropped B‐mode and SE‐mode ultrasound (US) lesion images to classify benign and malignant lesions. The MFF consists of an integrated feature network (IFN) and a decision network (DN). Unlike other recent fusion methods, the proposed MFF method can simultaneously learn complementary information from convolutional neural networks (CNNs) trained using B‐mode and SE‐mode US images. The features from the CNNs are ensembled using the multimodal EmbraceNet model and DN classifies the images using those features. The experimental results (sensitivity of 100 ± 0.00% and specificity of 94.28 ± 7.00%) on the real‐world clinical data showed that the proposed method outperforms the existing single‐ and multimodal methods. The proposed method predicts seven benign patients as benign three times out of five trials and six malignant patients as malignant five out of five trials. The proposed method would potentially enhance the classification accuracy of radiologists for breast cancer detection in US images.

show abstract

Section: Related Workmentioning

confidence: 99%

Deep learning‐based multimodal fusion network for segmentation and classification of breast cancers using B‐mode and elastography ultrasound images

Misra

Yoon

Kim

et al. 2022

Bioengineering & Transla Med

View full text Add to dashboard Cite

show abstract

“…Note that some multi-modal models are closely related to this task as well. We therefore select two state-of-the-art works: AW3M [8] and AdaMML [15], as the former treats different branches differently, while the latter selects different modalities to perform classification for different patients. For ablations study, we also implement the proposed model without the PAWN and VACL (row 7, Tab.…”

Section: Materials and Experimentsmentioning

confidence: 99%

Personalized Diagnostic Tool for Thyroid Cancer Classification using Multi-view Ultrasound

Huang¹,

Dong²,

Jia³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Over the past decades, the incidence of thyroid cancer has been increasing globally. Accurate and early diagnosis allows timely treatment and helps to avoid over-diagnosis. Clinically, a nodule is commonly evaluated from both transverse and longitudinal views using thyroid ultrasound. However, the appearance of the thyroid gland and lesions can vary dramatically across individuals. Identifying key diagnostic information from both views requires specialized expertise. Furthermore, finding an optimal way to integrate multi-view information also relies on the experience of clinicians and adds further difficulty to accurate diagnosis. To address these, we propose a personalized diagnostic tool that can customize its decision-making process for different patients. It consists of a multi-view classification module for feature extraction and a personalized weighting allocation network that generates optimal weighting for different views. It is also equipped with a self-supervised view-aware contrastive loss to further improve the model robustness towards different patient groups. Experimental results show that the proposed framework can better utilize multi-view information and outperform the competing methods.

show abstract

“…To address this issue, Zhang et al [37] proposed to use the Mean Squared Error (MSE) to align multimodal feature maps and designed a new contrastive loss to enforce the network to focus on the similarities of segmentation masks from paired modalities as well as dissimilarities of unpaired multi-modal data. Huang et al [38] proposed a SSL algorithm for four-modality ultrasound learning, where Mean Absolute Error across different modalities was minimized to ensure that high-level image features extracted from different modalities can be similar.…”

Section: B Ssl In Medical Imagingmentioning

confidence: 99%

Dual Sum-Product Networks Autoencoder for Multi-Label Classification

Wang

Zhang

Chen

2020

J. Shanghai Jiaotong Univ. (Sci.)

View full text Add to dashboard Cite

The clinical diagnosis of skin lesion involves the analysis of dermoscopic and clinical modalities. Dermoscopic images provide a detailed view of the surface structures whereas clinical images offer a complementary macroscopic information. The visual diagnosis of melanoma is also based on seven-point checklist which involves identifying different visual attributes. Recently, supervised learning approaches such as convolutional neural networks (CNNs) have shown great performances using both dermoscopic and clinical modalities (Multi-modality). The seven different visual attributes in the checklist are also used to further improve the the diagnosis. The performances of these approaches, however, are still reliant on the availability of large-scaled labeled data. The acquisition of annotated dataset is an expensive and timeconsuming task, more so with annotating multi-attributes.To overcome this limitation, we propose a self-supervised learning (SSL) algorithm for multi-modality skin lesion classification. Our algorithm enables the multi-modality learning by maximizing the similarities between paired dermoscopic and clinical images from different views. In addition, we generate surrogate pseudo-multi-labels that represent seven attributes via clustering analysis. We also propose a label-relation-aware module to refine each pseudo-label embedding and capture the interrelationships between pseudo-multi-labels. We validated the effectiveness of our algorithm using well-benchmarked seven-point skin lesion dataset. Our results show that our algorithm achieved better performances than other state-of-the-art SSL counterparts.

show abstract

AW3M: An auto-weighting and recovery framework for breast cancer diagnosis using multi-modal ultrasound

Cited by 32 publications

References 24 publications

Deep learning‐based multimodal fusion network for segmentation and classification of breast cancers using B‐mode and elastography ultrasound images

Deep learning‐based multimodal fusion network for segmentation and classification of breast cancers using B‐mode and elastography ultrasound images

Personalized Diagnostic Tool for Thyroid Cancer Classification using Multi-view Ultrasound

Dual Sum-Product Networks Autoencoder for Multi-Label Classification

Contact Info

Product

Resources

About