Contrastive-weighted self-supervised model for long-tailed data classification with vision transformer augmented

Hou, Rujie; Chen, Jinglong; Feng, Yong; Liu, Shen; He, Shuilong; Zhou, Zitong

doi:10.1016/j.ymssp.2022.109174

Cited by 39 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…A preliminary study investigating masked pretraining for transformers demonstrates its fine-tuning performance on a data-scarce, 4-way bearing classification task. [33] Contrastive pretraining was also considered for downstream finetuning on long-tailed fault distributions [34], and for learning more robust latent features [35]. These studies verify the use of pre-training for bearing classification; however, they pretrain and fine-tune on the same classes and datasets.…”

Section: Introductionmentioning

confidence: 90%

FaultFormer: Pretraining Transformers for Adaptable Bearing Fault Classification

Zhou,

Barati Farimani

2024

IEEE Access

View full text Add to dashboard Cite

The growth of global consumption has motivated important applications of deep learning to smart manufacturing and machine health monitoring. In particular, analyzing vibration data offers great potential to extract meaningful insights into predictive maintenance by the detection of bearing faults. Deep learning can be a powerful method to predict these mechanical failures; however, they lack generalizability to new tasks or datasets and require expensive, labeled mechanical data. We address this by presenting novel self-supervised pretraining and fine-tuning frameworks to identify bearing faults based on transformer models. In particular, we investigate different tokenization and data augmentation strategies to reach state-of-the-art accuracies using transformer models. Furthermore, we demonstrate self-supervised masked pretraining for vibration signals and its application to low-data regimes, task adaptation, and dataset adaptation. Pretraining is able to improve performance on 10-way bearing classification on scarce, unseen training samples. Transformer models also benefit from pretraining when fine-tuning on fault classes outside of the pretraining distribution. Lastly, pretrained transformers are shown to be able to generalize to a different dataset in a few-shot manner. This introduces a new paradigm where models can be pretrained on unlabeled data from different bearings, faults, and machinery and quickly deployed to new, data-scarce applications to suit specific manufacturing needs.

show abstract

Section: Introductionmentioning

confidence: 90%

FaultFormer: Pretraining Transformers for Adaptable Bearing Fault Classification

Zhou,

Barati Farimani

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Vision Transformer (ViT), which is based on self-attention, can capture long-distance dependencies in images and extract global features of images, and it is becoming a new direction in the field of computer vision [24]. ViT has been applied in various fields including industry [25], medicine [26] and agriculture [27] but currently has fewer applications in the field of livestock phenotypic measurements. The outstanding performance of ViT is based on a huge data size and sacrifices a large number of computational resources, making it difficult to apply to small datasets [28].…”

Section: Introductionmentioning

confidence: 99%

Non-Contact Measurement of Pregnant Sows’ Backfat Thickness Based on a Hybrid CNN-ViT Model

et al. 2023

Agriculture

View full text Add to dashboard Cite

Backfat thickness (BF) is closely related to the service life and reproductive performance of sows. The dynamic monitoring of sows’ BF is a critical part of the production process in large-scale pig farms. This study proposed the application of a hybrid CNN-ViT (Vision Transformer, ViT) model for measuring sows’ BF to address the problems of high measurement intensity caused by the traditional contact measurement of sows’ BF and the low efficiency of existing non-contact models for measuring sows’ BF. The CNN-ViT introduced depth-separable convolution and lightweight self-attention, mainly consisting of a Pre-local Unit (PLU), a Lightweight ViT (LViT) and an Inverted Residual Unit (IRU). This model could extract local and global features of images, making it more suitable for small datasets. The model was tested on 106 pregnant sows with seven randomly divided datasets. The results showed that the CNN-ViT had a Mean Absolute Error (MAE) of 0.83 mm, a Root Mean Square Error (RMSE) of 1.05 mm, a Mean Absolute Percentage Error (MAPE) of 4.87% and a coefficient of determination (R-Square, R2) of 0.74. Compared to LviT-IRU, PLU-IRU and PLU-LviT, the CNN-ViT’s MAE decreased by more than 12%, RMSE decreased by more than 15%, MAPE decreased by more than 15% and R² improved by more than 17%. Compared to the Resnet50 and ViT, the CNN-ViT’s MAE decreased by more than 7%, RMSE decreased by more than 13%, MAPE decreased by more than 7% and R2 improved by more than 15%. The method could better meet the demand for the non-contact automatic measurement of pregnant sows’ BF in actual production and provide technical support for the intelligent management of pregnant sows.

show abstract

“…Essas abordagens enfrentam consideráveis quedas de eficácia quando avaliadas sob as perspectivas do desbalanceamento e da qualidade, especialmente em relac ¸ão aos rótulos menos frequentes, doravante referidos como rótulos tail. De fato, os rótulos tail correspondem a cerca de 80% do espac ¸o de rótulos em vários cenários do mundo real [Hou et al 2022, Ge et al 2022, Huang and Wu 2019.…”

Section: Introduc ¸ãOunclassified

Repense, Recupere e Reranqueie: Um Pipeline de Recuperação e Reranqueamento para Classificação de Texto Multi-rótulo Extrema

França,

Ribeiro-Neto,

Gonçalves

2023

Anais Estendidos Do XXXVIII Simpósio Brasileiro De Banco De Dados (SBBD Estendido 2023)

View full text Add to dashboard Cite

Extreme multi-label text classification (XMTC) involves assigning relevant labels to text from a huge space of labels. Addressing the core challenges of XMTC (volume, imbalance and quality), we propose xCoRetriev, a two-stage pipeline migrating from a classification perspective to an information retrieval (IR) approach. We address the volume challenge by efficiently combining IR methods; we face the imbalance challenge by better capturing the text-label relationship and; we've improved quality by enriching tag names with pseudo-labels. Our results demonstrate the strengths of xCoRetriev when compared to baselines in terms of: (i) scalability for large spaces of labels and amount of text; (ii) effectiveness in the face of high imbalance, especially for predicting infrequent labels – with gains of up to 40% in MRR and NDCG –; and (iii) ability to handle poor quality annotated text and labels.

show abstract

Contrastive-weighted self-supervised model for long-tailed data classification with vision transformer augmented

Cited by 39 publications

References 22 publications

FaultFormer: Pretraining Transformers for Adaptable Bearing Fault Classification

FaultFormer: Pretraining Transformers for Adaptable Bearing Fault Classification

Non-Contact Measurement of Pregnant Sows’ Backfat Thickness Based on a Hybrid CNN-ViT Model

Repense, Recupere e Reranqueie: Um Pipeline de Recuperação e Reranqueamento para Classificação de Texto Multi-rótulo Extrema

Contact Info

Product

Resources

About