Text Sentiment Classification Based on BERT Embedding and Sliced Multi-Head Self-Attention Bi-GRU

Zhang, Xiangsen; Wu, Zhongqiang; Liu, Ke; Zhao, Zhihua; Wang, Jinhao; Wu, Chengqin

doi:10.3390/s23031481

Cited by 31 publications

(10 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This approach enables a deeper contextual understanding of the data, crucial for handling the complex nature of protein sequences. Additionally, the model incorporates a multi-head attention mechanism, allowing it to capture diverse contextual representations from multiple perspectives. This mechanism divides the input into multiple heads, with each head focusing on different parts of the sequence, thus enabling a more nuanced and comprehensive understanding of the protein sequences .…”

Section: Methodsmentioning

confidence: 99%

MucLiPred: Multi-Level Contrastive Learning for Predicting Nucleic Acid Binding Residues of Proteins

Zhang,

Wang,

Wei

2024

J. Chem. Inf. Model.

View full text Add to dashboard Cite

Protein−molecule interactions play a crucial role in various biological functions, with their accurate prediction being pivotal for drug discovery and design processes. Traditional methods for predicting protein−molecule interactions are limited. Some can only predict interactions with a specific molecule, restricting their applicability, while others target multiple molecule types but fail to efficiently process diverse interaction information, leading to complexity and inefficiency. This study presents a novel deep learning model, MucLiPred, equipped with a dual contrastive learning mechanism aimed at improving the prediction of multiple molecule-protein interactions and the identification of potential molecule-binding residues. The residue-level paradigm focuses on differentiating binding from non-binding residues, illuminating detailed local interactions. The type-level paradigm, meanwhile, analyzes overarching contexts of molecule types, like DNA or RNA, ensuring that representations of identical molecule types gravitate closer in the representational space, bolstering the model's proficiency in discerning interaction motifs. This dual approach enables comprehensive multi-molecule predictions, elucidating the relationships among different molecule types and strengthening precise protein−molecule interaction predictions. Empirical evidence demonstrates MucLiPred's superiority over existing models in robustness and prediction accuracy. The integration of dual contrastive learning techniques amplifies its capability to detect potential molecule-binding residues with precision. Further optimization, separating representational and classification tasks, has markedly improved its performance. MucLiPred thus represents a significant advancement in protein−molecule interaction prediction, setting a new precedent for future research in this field.

show abstract

Section: Methodsmentioning

confidence: 99%

MucLiPred: Multi-Level Contrastive Learning for Predicting Nucleic Acid Binding Residues of Proteins

Zhang,

Wang,

Wei

2024

J. Chem. Inf. Model.

View full text Add to dashboard Cite

show abstract

“…Other metrics such as SPICE [2] account for such variety by comparing parse trees, however, they still can not account for different word choice. Inspired by advances in language models as well as approaches like BERTScore [110] and CLIPScore [40], we use a pretrained language model to compute caption similarity.…”

Section: Sampling Image Pairs Using Languagementioning

confidence: 99%

Learning Visual Representations via Language-Guided Sampling

Banani¹,

Desai²,

Johnson³

2023

Preprint

View full text Add to dashboard Cite

Although an object may appear in numerous contexts, we often describe it in a limited number of ways. This happens because language abstracts away visual variation to represent and communicate concepts. Building on this intuition, we propose an alternative approach to visual learning: using language similarity to sample semantically similar image pairs for contrastive learning. Our approach deviates from image-based contrastive learning by using language to sample pairs instead of hand-crafted augmentations or learned clusters. Our approach also deviates from image-text contrastive learning by relying on pre-trained language models to guide the learning rather than minimize a cross-modal similarity. Through a series of experiments, we show that language-guided learning can learn better features than both image-image and image-text representation learning approaches.

show abstract

“…Overall, the proposed LeBERT model was 88.2% accurate when used for binary classification. Zhang et al [21] presented a Sliced BI-GRU (bidirectional-gated recurrent unit) architecture that employs BERT embedding in conjunction with the multi-head self-attention mechanism. The BERT models' word vector representation first, which plays a role in the neural network's embedding layer, and then they divide the input sequence into equal-length chunks.…”

Section: Literature Reviewmentioning

confidence: 99%

Transfer Learning for Sentiment Classification Using Bidirectional Encoder Representations from Transformers (BERT) Model

Areshey

Mathkour

2023

Sensors

View full text Add to dashboard Cite

Sentiment is currently one of the most emerging areas of research due to the large amount of web content coming from social networking websites. Sentiment analysis is a crucial process for recommending systems for most people. Generally, the purpose of sentiment analysis is to determine an author’s attitude toward a subject or the overall tone of a document. There is a huge collection of studies that make an effort to predict how useful online reviews will be and have produced conflicting results on the efficacy of different methodologies. Furthermore, many of the current solutions employ manual feature generation and conventional shallow learning methods, which restrict generalization. As a result, the goal of this research is to develop a general approach using transfer learning by applying the “BERT (Bidirectional Encoder Representations from Transformers)”-based model. The efficiency of BERT classification is then evaluated by comparing it with similar machine learning techniques. In the experimental evaluation, the proposed model demonstrated superior performance in terms of outstanding prediction and high accuracy compared to earlier research. Comparative tests conducted on positive and negative Yelp reviews reveal that fine-tuned BERT classification performs better than other approaches. In addition, it is observed that BERT classifiers using batch size and sequence length significantly affect classification performance.

show abstract

Text Sentiment Classification Based on BERT Embedding and Sliced Multi-Head Self-Attention Bi-GRU

Cited by 31 publications

References 33 publications

MucLiPred: Multi-Level Contrastive Learning for Predicting Nucleic Acid Binding Residues of Proteins

MucLiPred: Multi-Level Contrastive Learning for Predicting Nucleic Acid Binding Residues of Proteins

Learning Visual Representations via Language-Guided Sampling

Transfer Learning for Sentiment Classification Using Bidirectional Encoder Representations from Transformers (BERT) Model

Contact Info

Product

Resources

About