Multi-Modal Multi-Scale Deep Learning for Large-Scale Image Annotation

Niu, Yulei; Lu, Zhiwu; Wen, Ji-Rong; Xiang, Tao; Chang, Shih-Fu

doi:10.1109/tip.2018.2881928

Cited by 75 publications

(43 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many researchers have pointed out that the per-word metrics are biased toward infrequent labels, because making them correct could have a very significant impact on final accuracy [24]. Therefore they propose overall metrics (sometimes called as per-image metrics) [24], [29]- [31]. The overall metrics are defined as,…”

Section: B Evaluation Metricsmentioning

confidence: 99%

“…In essence, as far as image understanding is concerned, the overall metrics are much better than per-word metrics, since the overall metrics pay more attention to image content understanding in the perspective of each image. As a result, in common with per-word metrics, the overall metrics are considered as appropriate metrics for comparing image annotation methods [24], [29]- [31]. In addition, we also introduce hybrid F1-measure (called H-F1) combining per-word F1-measure and overall F1-measure with the harmonic mean [29].…”

Section: E Further Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

A Weighted Topic Model Learned From Local Semantic Space for Automatic Image Annotation

et al. 2020

View full text Add to dashboard Cite

Automatic image annotation plays a significant role in image understanding, retrieval, classification, and indexing. Today, it is becoming increasingly important in order to annotate large-scale social media images from content-sharing websites and social networks. These social images are usually annotated by user-provided low-quality tags. The topic model is considered as a promising method to describe these weak-labeling images by learning latent representations of training samples. The recent annotation methods based on topic models have two shortcomings. First, they are difficult to scale to a large-scale image dataset. Second, they can not be used to online image repository because of continuous addition of new images and new tags. In this paper, we propose a novel annotation method based on topic model, namely local learning-based probabilistic latent semantic analysis (LL-PLSA), to solve the above problems. The key idea is to train a weighted topic model for a given test image on its semantic neighborhood consisting of a fixed number of semantically and visually similar images. This method can scale to a large-scale image database, as training samples involved in modeling are a few nearest neighbors rather than the entire database. Moreover, this proposed topic model, online customized for the test image, naturally addresses the issue of continuous addition of new images and new tags in a database. Extensive experiments on three benchmark datasets demonstrate that the proposed method significantly outperforms the state-of-the-art especially in terms of overall metrics. INDEX TERMS Automatic image annotation, image retrieval, probabilistic latent semantic analysis, topic model.

show abstract

Section: B Evaluation Metricsmentioning

confidence: 99%

Section: E Further Evaluationmentioning

confidence: 99%

A Weighted Topic Model Learned From Local Semantic Space for Automatic Image Annotation

et al. 2020

View full text Add to dashboard Cite

show abstract

“…), similar to a previous approach. 16 Second, we provide the model with more power by adding an FC layer after concatenation which allows the model to learn a nonlinear combination of the features (Dense-GRU FC). Third, we do not combine the features explicitly but instead let them learn interactions between each other by employing a cross-attention module (Dense-GRU CA).…”

Section: Modelsmentioning

confidence: 99%

Melanoma detection with electrical impedance spectroscopy and dermoscopy using joint deep learning models

Gessert

Bengs

Schlaefer

2020

Medical Imaging 2020: Computer-Aided Diagnosis

View full text Add to dashboard Cite

The initial assessment of skin lesions is typically based on dermoscopic images. As this is a difficult and timeconsuming task, machine learning methods using dermoscopic images have been proposed to assist human experts. Other approaches have studied electrical impedance spectroscopy (EIS) as a basis for clinical decision support systems. Both methods represent different ways of measuring skin lesion properties as dermoscopy relies on visible light and EIS uses electric currents. Thus, the two methods might carry complementary features for lesion classification. Therefore, we propose joint deep learning models considering both EIS and dermoscopy for melanoma detection. For this purpose, we first study machine learning methods for EIS that incorporate domain knowledge and previously used heuristics into the design process. As a result, we propose a recurrent model with state-max-pooling which automatically learns the relevance of different EIS measurements. Second, we combine this new model with different convolutional neural networks that process dermoscopic images. We study ensembling approaches and also propose a cross-attention module guiding information exchange between the EIS and dermoscopy model. In general, combinations of EIS and dermoscopy clearly outperform models that only use either EIS or dermoscopy. We show that our attention-based, combined model outperforms other models with specificities of 34.4 % ) and 53.7 % (CI 50.1-57.6) for dermoscopy, EIS and the combined model, respectively, at a clinically relevant sensitivity of 98 %.Recently, other automatic diagnosis methods have been proposed using dermoscopic images as an input to convolutional neural networks (CNNs). 7, 8 Also, the ISIC 2018 Skin Lesion Analysis Towards Melanoma Detection Challenge 9 resulted in numerous deep learning-based methods for high-performance skin lesion classification. 10 So far, machine learning methods for EIS and dermoscopy have been addressed separately, although dermoscopic images are generally acquired before EIS measurements. 3 EIS and dermoscopy represent very different ways of measuring skin lesion properties. While dermoscopy captures light absorption mostly at the skin surface, EIS provides electrical resistance properties in deeper skin layers. Thus, the two methods might complement each

show abstract

“…ese solutions treat the image annotation problem as an image-to-text translation problem and solve it using an encoder-decoder model. e multiscale approaches [18] propose a novel multiscale deep model for extracting rich and discriminative features capable of representing a wide range of visual concepts. Instead of CNN features, some works use more semantic information obtained from the image as the input to the decoder [19,20].…”

Section: Image Taggingmentioning

confidence: 99%

Image-Text Joint Learning for Social Images with Spatial Relation Model

Feng

Zhou

et al. 2020

Complexity

View full text Add to dashboard Cite

e rapid developments in sensor technology and mobile devices bring a flourish of social images, and large-scale social images have attracted increasing attention to researchers. Existing approaches generally rely on recognizing object instances individually with geo-tags, visual patterns, etc. However, the social image represents a web of interconnected relations; these relations between entities carry semantic meaning and help a viewer differentiate between instances of a substance. is article forms the perspective of the spatial relationship to exploring the joint learning of social images. Precisely, the model consists of three parts: (a) a module for deep semantic understanding of images based on residual network (ResNet); (b) a deep semantic analysis module of text beyond traditional word bag methods; (c) a joint reasoning module from which the text weights obtained using image features on self-attention and a novel tree-based clustering algorithm. e experimental results demonstrate the effectiveness of using Flickr30k and Microsoft COCO datasets. Meanwhile, our method considers spatial relations while matching.

show abstract

Multi-Modal Multi-Scale Deep Learning for Large-Scale Image Annotation

Cited by 75 publications

References 43 publications

A Weighted Topic Model Learned From Local Semantic Space for Automatic Image Annotation

A Weighted Topic Model Learned From Local Semantic Space for Automatic Image Annotation

Melanoma detection with electrical impedance spectroscopy and dermoscopy using joint deep learning models

Image-Text Joint Learning for Social Images with Spatial Relation Model

Contact Info

Product

Resources

About