Deep learning‐based multimodal fusion network for segmentation and classification of breast cancers using B‐mode and elastography ultrasound images

Abstract

Ultrasonography is one of the key medical imaging modalities for evaluating breast lesions. For differentiating benign from malignant lesions, computer‐aided diagnosis (CAD) systems have greatly assisted radiologists by automatically segmenting and identifying features of lesions. Here, we present deep learning (DL)‐based methods to segment the lesions and then classify benign from malignant, utilizing both B‐mode and strain elastography (SE‐mode) images. We propose a weighted multimodal U‐Net (W‐MM‐U‐Net) model for segmenting lesions where optimum weight is assigned on different imaging modalities using a weighted‐skip connection method to emphasize its importance. We design a multimodal fusion framework (MFF) on cropped B‐mode and SE‐mode ultrasound (US) lesion images to classify benign and malignant lesions. The MFF consists of an integrated feature network (IFN) and a decision network (DN). Unlike other recent fusion methods, the proposed MFF method can simultaneously learn complementary information from convolutional neural networks (CNNs) trained using B‐mode and SE‐mode US images. The features from the CNNs are ensembled using the multimodal EmbraceNet model and DN classifies the images using those features. The experimental results (sensitivity of 100 ± 0.00% and specificity of 94.28 ± 7.00%) on the real‐world clinical data showed that the proposed method outperforms the existing single‐ and multimodal methods. The proposed method predicts seven benign patients as benign three times out of five trials and six malignant patients as malignant five out of five trials. The proposed method would potentially enhance the classification accuracy of radiologists for breast cancer detection in US images.

A twin convolutional neural network with hybrid binary optimizer for multimodal breast cancer digital image classification

“…The framework combines a decision network and an integrated feature network to learn multimodal features from B-mode and strain elastography mode when ultrasound images are supplied to multiple CNNs. The study uses multimodal EmbraceNet to fuse the features learnt using the CNN models 35 . A two-level machine learning technique had been applied to separately extract unimodal information from two different images sources to classify and then predict breast cancer.…”

“…Similarly, the use of a weakly supervised clustering-constrained attention multiple instance learning (CLAM) method has been used to overcome the problem of insufficient data in multimodal feature fusion 34 . Multimodal fusion framework (MFF) which relies on a weighted multimodal U-Net (WMM-UNet) have also been proposed with EmbraceNet used to ensemble multimodal features 35 . Another approach to the use of transfer learning in multimodal fusion learning was demonstrated using social engineering optimization with deep transfer learning on photoacoustic multimodal imaging (PAMI) 36 .…”

Advanced technologies for biomedical applications by emerging researchers in Asia‐Pacific

“…28 Misra et al introduced novel deep learning-based techniques for segmenting lesions and distinguishing between benign and malignant cases by analyzing B-mode and strain elastography images. 29 In another research direction, Yuan et al utilized machine learning methods to predict the permeation of drugs through the microneedletreated skin. 30…”

“…Tostado et al developed an AI‐assisted single‐cell phenomic‐transcriptomic platform, which showed great potential in elucidating mechanisms and therapeutic targets against immune response 28 . Misra et al introduced novel deep learning‐based techniques for segmenting lesions and distinguishing between benign and malignant cases by analyzing B‐mode and strain elastography images 29 . In another research direction, Yuan et al utilized machine learning methods to predict the permeation of drugs through the microneedle‐treated skin 30 …”

A residual-attention multimodal fusion network (ResAMF-Net) for detection and classification of breast cancer

“…Recent CAD studies utilize various combination of modalities, including gene expression profile, digital pathology, MRI, ultrasound images, DBT, FFDM [1][2][3][4][5][6]. These multimodal fusion frameworks can be divided into pixel fusion stage, feature fusion and decision fusion.…”