Cross-modal fusion for multi-label image classification with attention mechanism

Wang, Yangtao; Xie, Yanzhao; Zeng, Jiangfeng; Wang, Hanpin; Fan, Lisheng; Song, Yufan

doi:10.1016/j.compeleceng.2022.108002

Cited by 17 publications

(6 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The attention mechanism in deep learning simulates the way human attention works, focusing on more important regional information and reducing the interference from unimportant information. In the process of image processing, more attention is paid to regions that match specific features, thus achieving efficient and rational allocation of computational resources [29,30]. Currently, attention mechanisms have been widely applied in object detection models.…”

Section: Simam Attention Modulementioning

confidence: 99%

Small-modulus worms surface defect detection method based on YOLOv7

Li,

Zheng,

et al. 2024

Meas. Sci. Technol.

View full text Add to dashboard Cite

Small-modulus worms are widely used in precision transmission mechanisms. The detection of surface defects on processed Small-modulus worms mainly relies on manual inspection. However, this detection method has low detection efficiency and low accuracy. In response to this issue, this paper proposed a surface defect detection method for small-modulus worms based on deep learning. Firstly, based on the geometric features and material properties of small-modulus worms, designed a two-cameras worms image acquisition system to capture small-modulus worm images. And the captured images were annotated and classified to construct a small-modulus worms defect dataset. Secondly, the YOLOv7 network model algorithm was studied and improved. A three-stage image preprocessing algorithm was added to the YOLOV7 model to reduce the interference in distorted areas. Next, Ghost-CBS structures were employed to replace some of the conventional CBS structures, reducing computational complexity. And we also added a width-height balance weights and alpha structure to the eiou loss function. Improve the fitting performance of the model on the bounding box with large aspect ratio. Furthermore, three SimAM attention modules were added to the backbone, increasing the network's focus on key areas. Finally, experimental validation was conducted, and the results demonstrated that the performance of our proposed method is better than other existing methods. The detection accuracy reached 95.7%, with a map@50 of 94.6%. Overall, the performance met the requirements for worm defect detection.

show abstract

Section: Simam Attention Modulementioning

confidence: 99%

Small-modulus worms surface defect detection method based on YOLOv7

Li,

Zheng,

et al. 2024

Meas. Sci. Technol.

View full text Add to dashboard Cite

show abstract

“…Cross-modal attention mechanism methods can be realized by dot product attention, bilinear attention, or multi-head attention, etc. [14]. 4) Cross-modal Pre-training Methods: By pre-training on large-scale multimodal data, cross-modal representation capabilities are learned and then fine-tuned on specific tasks.…”

Section: Related Work 21 Sentiment Analysis Of Multimodal Datamentioning

confidence: 99%

Emotion Intensity Detection in Online Media: An Attention Mechanism Based Multimodal Deep Learning Approach

2024

Teh. vjesn.

View full text Add to dashboard Cite

With the increasing influence of online public opinion, mining opinions and trend analysis from massive data of online media is important for understanding user sentiment, managing brand reputation, analyzing public opinion and optimizing marketing strategies. By combining data from multiple perceptual modalities, more comprehensive and accurate sentiment analysis results can be obtained. However, using multimodal data for sentiment analysis may face challenges such as data fusion, modal imbalance and inter-modal correlation. To overcome these challenges, the paper introduces an attention mechanism to multimodal sentiment analysis by constructing text, image, and audio feature extractors and using a custom cross-modal attention layer to compute the attention weights between different modalities, and finally fusing the attention-weighted features for sentiment classification. Through the cross-modal attention mechanism, the model can automatically learn the correlation between different modalities, dynamically adjust the modal weights, and selectively fuse features from different modalities, thus improving the accuracy and expressiveness of sentiment analysis.

show abstract

“…When processing images, it can pay more attention to image regions that can match target features and ignore background regions that cannot match. In other words, it can achieve a reasonable allocation of information resources [26,27]. Current attention mechanisms are more widely used in CNN architectures, including the channel attention mechanism, SENet [28] and the spatial attention mechanism, CBAM [29].…”

Section: Optimization At the Stage Of Feature Outputmentioning

confidence: 99%

An efficient method of pavement distress detection based on improved YOLOv7

Xiao

et al. 2023

Meas. Sci. Technol.

View full text Add to dashboard Cite

Pavements play a pivotal role in infrastructure construction, so Pavement Distress Detection (PDD) will greatly affect pavement service life and vehicle operation safety. Traditional manual detection and computer detection methods have such disadvantages as low efficiency, high cost and error-proneness. Thus, they are not suitable for high-speed detection tasks with a large number of defects. The defect detection method based on deep learning can achieve end-to-end target detection, generalize and monitor targets in real time. On such basis, this paper has proposed an efficient method of PDD based on improved YOLOv7. YOLOv7, which is the best-performing object detection model in the YOLO series, is known for its high efficiency, strong scalability, and support for panoramic detection. It lays a solid foundation for enhancing PDD models. In this paper, this model will be improved from model speed and accuracy. Firstly, SimAM attention module is employed to weight feature images, which has greatly improved model accuracy. Secondly, Ghost module in place of partial deep convolution module is used to improve model running speed. Then SIoU, instead of the original localization loss function, is performed to optimize model training process. Finally, the proposed improved YOLOv7 model is applied to different road defect datasets and compared with other methods, such as Faster R-CNN, CenterNet, DETR, YOLOv6 and the original YOLOv7 model. The results show that the proposed method has ubiquitous advantages than the above-mentioned methods, with the average mAP, F1 value and FPS value of 85.8%, 0.697 and 62.13fps respectively. Furthermore, the values of the parameters Params and FLOPs also decrease to some degree.

show abstract

Cross-modal fusion for multi-label image classification with attention mechanism

Cited by 17 publications

References 16 publications

Small-modulus worms surface defect detection method based on YOLOv7

Small-modulus worms surface defect detection method based on YOLOv7

Emotion Intensity Detection in Online Media: An Attention Mechanism Based Multimodal Deep Learning Approach

An efficient method of pavement distress detection based on improved YOLOv7

Contact Info

Product

Resources

About