Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey

Jing, Longlong; Tian, Yingli

doi:10.1109/tpami.2020.2992393

Cited by 1,447 publications

(846 citation statements)

References 126 publications

Supporting

Mentioning

843

Contrasting

Order By: Relevance

“…Features extracted from pre-trained SSL models are known as Self Supervised Embeddings (SSE). SSL has become a prominent representation learning paradigm in both Natural Language Processing (NLP) and Computer Vision (CV) communities [15], [20], [26]. SSL algorithms have two stages.…”

Section: B Multimodal Features Extracted From Pre-trained-ssl Algorimentioning

confidence: 99%

“…In contrast to previous work, we represent all input modalities (audio, video, and text) with deep features extracted from pre-trained Self Supervised Learning (SSL) -a powerful representation learning technique -models [15]- [17]. Although SSL features give powerful representations of the input modalities, it is an extremely challenging task to fuse them before the final prediction due to the following reasons: 1) High dimensionality of SSL embeddings 2) Longer sequence lengths of SSL features 3) Mismatch between sizes and sequence lengths of SSL features across modalities that have extracted from different SSL models Although a simple concatenation seems like a viable option, additional trainable parameters required to fully connect the high dimensional SSL embeddings make the network FIGURE 1: Overview of the Self Supervised Embedding Fusion Transformer (SSE-FT)…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion

et al. 2020

View full text Add to dashboard Cite

Emotion Recognition is a challenging research area given its complex nature, and humans express emotional cues across various modalities such as language, facial expressions, and speech. Representation and fusion of features are the most crucial tasks in multimodal emotion recognition research. Self Supervised Learning (SSL) has become a prominent and influential research direction in representation learning, where researchers have access to pre-trained SSL models that represent different data modalities. For the first time in the literature, we represent three input modalities of text, audio (speech), and vision with features extracted from independently pre-trained SSL models in this paper. Given the high dimensional nature of SSL features, we introduce a novel Transformers and Attention-based fusion mechanism that can combine multimodal SSL features and achieve state-of-the-art results for the task of multimodal emotion recognition. We benchmark and evaluate our work to show that our model is robust and outperforms the state-of-the-art models on four datasets.

show abstract

Section: B Multimodal Features Extracted From Pre-trained-ssl Algorimentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Voxelization representation is widely used in 3D modeling tasks and other related 3D vision tasks. For example, VoxelNet [ 17 ], VConv-DAE [ 18 ], and LightNet [ 19 , 20 , 21 ].…”

Section: Related Workmentioning

confidence: 99%

3DMGNet: 3D Model Generation Network Based on Multi-Modal Data Constraints and Multi-Level Feature Fusion

Wang

Xue

et al. 2020

Sensors

View full text Add to dashboard Cite

Due to the limitation of less information in a single image, it is very difficult to generate a high-precision 3D model based on the image. There are some problems in the generation of 3D voxel models, e.g., the information loss at the upper level of a network. To solve these problems, we design a 3D model generation network based on multi-modal data constraints and multi-level feature fusion, named as 3DMGNet. Moreover, 3DMGNet is trained by self-supervised method to achieve 3D voxel model generation from an image. The image feature extraction network (2DNet) and 3D feature extraction network (3D auxiliary network) are used to extract the features of the image and 3D voxel model. Then, feature fusion is used to integrate the low-level features into the high-level features in the 3D auxiliary network. To extract more effective features, each layer of the feature map in feature extraction network is processed by an attention network. Finally, the extracted features generate 3D models by a 3D deconvolution network. The feature extraction of 3D model and the generation of voxelization play an auxiliary role in the training of the whole network for the 3D model generation based on an image. Additionally, a multi-view contour constraint method is proposed, to enhance the effect of the 3D model generation. In the experiment, the ShapeNet dataset is adapted to prove the effect of the 3DMGNet, which verifies the robust performance of the proposed method.

show abstract

“…Deep learning methods, particularly deep convolutional neural networks (CNN) [20,21], have quickly become the preferred approach for processing medical images [22,23]. Large-scale datasets are usually required to train deep neural networks [24]. The ChestX-ray 14 dataset, released by the National Institutes of Health (NIH) in 2017 [25], is known as one of the largest hospital-scale ChestX-ray datasets.…”

Section: Introductionmentioning

confidence: 99%

Fusion High-Resolution Network for Diagnosing ChestX-ray Images

Huang

Lin

et al. 2020

Electronics

View full text Add to dashboard Cite

The application of deep convolutional neural networks (CNN) in the field of medical image processing has attracted extensive attention and demonstrated remarkable progress. An increasing number of deep learning methods have been devoted to classifying ChestX-ray (CXR) images, and most of the existing deep learning methods are based on classic pretrained models, trained by global ChestX-ray images. In this paper, we are interested in diagnosing ChestX-ray images using our proposed Fusion High-Resolution Network (FHRNet). The FHRNet concatenates the global average pooling layers of the global and local feature extractors—it consists of three branch convolutional neural networks and is fine-tuned for thorax disease classification. Compared with the results of other available methods, our experimental results showed that the proposed model yields a better disease classification performance for the ChestX-ray 14 dataset, according to the receiver operating characteristic curve and area-under-the-curve score. An ablation study further confirmed the effectiveness of the global and local branch networks in improving the classification accuracy of thorax diseases.

show abstract

Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey

Cited by 1,447 publications

References 126 publications

Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion

Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion

3DMGNet: 3D Model Generation Network Based on Multi-Modal Data Constraints and Multi-Level Feature Fusion

Fusion High-Resolution Network for Diagnosing ChestX-ray Images

Contact Info

Product

Resources

About