A Joint Representation Learning Approach for Social Media Tag Recommendation

Li, Xiangyu; Chen, Weizheng

doi:10.1007/978-3-030-92273-3_9

Cited by 4 publications

(7 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Protein-Text Contrastive Learning (PTC) aims to align the feature space of the protein encoder and the text decoder, encouraging parallel protein-text pairs have higher similarity scores. This objective has been demonstrated to be effective in ALBEF [32] in image-text learning task.…”

Section: Methodsmentioning

confidence: 99%

“…Protein-Text Matching (PTM) predicts whether a pair of protein and text is positive (matched) or negative (not matched). We follow the negative sampling strategy [32] , where negative pairs with higher contrastive similarity in a batch have a higher chance to be sampled. It aims to learn protein-text multimodal representation for protein-text pair that share similar semantics but differ in fine-grained details.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

FAPM: Functional Annotation of Proteins using Multi-Modal Models Beyond Structural Modeling

Xiang,

Xiong,

Chen

et al. 2024

Preprint

View full text Add to dashboard Cite

Assigning appropriate property labels, such as functional terms and catalytic activity, to proteins, remains a significant challenge, particularly for the non-homologous ones. In contrast to prior approaches that mostly focused on protein sequence features, we employ pretrained protein language model to encode the sequence features, and natural language model for the semantic information of property descriptions. Specifically, we present FAPM, a contrastive model between natural language and protein sequence language, which combines the pretrained protein sequence model with the pretrained large language model to generate labels such as GO functional terms and catalytic activity prediction in the format of natural language. Our result shows that FAPM has learned superior representations related to protein properties, outperforming protein sequence-based or structure-based models. Our model achieves state-of-the-art performance both on public benchmarks and on experimentally-annotated phage proteins that have limited known homologous sequences. Furthermore, we demonstrate the flexibility of our model to accept additional free text prompts as input, such as easily accessible taxonomy information, not only enhances its predictive performance but also improves the model's explainability. Our methodology presents a novel avenue for exploration, characterized by its potential to supersede current methodologies reliant on multiple sequence alignment for protein annotation.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

FAPM: Functional Annotation of Proteins using Multi-Modal Models Beyond Structural Modeling

Xiang,

Xiong,

Chen

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Four TF blocks are used, one for each of the unimodal and multimodal processing of each modality. In the multimodal blocks, pairwise cross-attention is added as means of communication between modalities, similar to [29,28,65], to ensure the coordination of the modalities. The model is trained using multiple supervised losses for the unimodal and multimodal predictions, and a self-supervised objective for aligning the unimodal representations, following previous works in vision and text [49,29].…”

Section: Our Contributionmentioning

confidence: 99%

“…Most works follow the general framework of aggregating information from one part of the network and transmitting it in a compressed form to the others. This can achieved for example with the summation of intermediate CNN channels [20], the exchange of intermediate representations [62], any of the Squeeze and Excite gates [34] or cross-attention weights [32,59,29,28,65]. The latter approach will also be exploited by CoRe-Sleep.…”

Section: Introductionmentioning

confidence: 99%

CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities

Kontras,

Chatzichristos,

Phan

et al. 2024

IEEE Trans. Neural Syst. Rehabil. Eng.

View full text Add to dashboard Cite

Sleep abnormalities can have severe health consequences. Automated sleep staging, i.e. labelling the sequence of sleep stages from the patient's physiological recordings, could simplify the diagnostic process. Previous work on automated sleep staging has achieved great results, mainly relying on the EEG signal. However, often multiple sources of information are available beyond EEG. This can be particularly beneficial when the EEG recordings are noisy or even missing completely. In this paper, we propose CoRe-Sleep, a Coordinated Representation multimodal fusion network that is particularly focused on improving the robustness of signal analysis on imperfect data. We demonstrate how appropriately handling multimodal information can be the key to achieving such robustness. CoRe-Sleep tolerates noisy or missing modalities segments, allowing training on incomplete data. Additionally, it shows state-of-the-art performance when testing on both multimodal and unimodal data using a single model on SHHS-1, the largest publicly available study that includes sleep stage labels. The results indicate that training the model on multimodal data does positively influence performance when tested on unimodal data. This work aims at bridging the gap between automated analysis tools and their clinical utility.

show abstract

“…Owens et al [12] studied a self supervised multi-sensory training method that automatically aligns audio and video. Li et al [13] used the methods of pre fusion alignment and momentum distillation to maximize mutual information, so as to make the modes correspond to each other as far as possible to achieve the goal of resource balance. Wang et al [14] used a gradient mixing method to solve the problem of varying generalization speeds for different modal data.…”

Section: Related Workmentioning

confidence: 99%

A multimodal resource balancing approach based on cross-attention and bidrectional associative memory networks

Wang,

Cheng,

Liu

et al. 2023

Third International Conference on Green Communication, Network, and Internet of Things (CNIoT 2023)

View full text Add to dashboard Cite

In the event of a disaster, mining social media tweets containing disaster information to analyze the dynamics of the disaster can help relevant authorities make quick emergency decisions and public opinion analysis. The data on social media jointly describe a thing to make it semantically related, but there are often structural and semantic imbalances between heterogeneous modalities. Most of the current research is devoted to formal complementation or semantic balancing, and multi-granularity resource balancing is beneficial to better obtain consistent information of heterogeneous modalities and remove redundancy. Based on the above problems, this paper proposes an end-to-end Multimodal Resource Balancing (MMRB) model, which designs cross-attention and two-way associative memory network modules while avoiding the loss of important information of unimodal modalities. The cross-attention module is used to capture the deep semantic correlations between modalities and weight the modal features to achieve semantic-level resource balancing. The associative memory module uses each modal feature to generate a complete feature representation of another modality on the pre-trained migration model to complement multimodal information at the feature level to improve the modal imbalance and modal missing problem of unstructured social media data. Finally, the fused features are weighted by the joint feature codec to reasonably focus on the contribution of different modal features in the joint features and the intrinsic association. Experiments are conducted on CrisisMMD, a public data of social media graphics used for disaster detection, and the accuracy of the model is stronger than the current multimodal resource balancing model, which verifies the effectiveness of our model.

show abstract

A Joint Representation Learning Approach for Social Media Tag Recommendation

Cited by 4 publications

References 16 publications

FAPM: Functional Annotation of Proteins using Multi-Modal Models Beyond Structural Modeling

FAPM: Functional Annotation of Proteins using Multi-Modal Models Beyond Structural Modeling

CoRe-Sleep: A Multimodal Fusion Framework for Time Series Robust to Imperfect Modalities

A multimodal resource balancing approach based on cross-attention and bidrectional associative memory networks

Contact Info

Product

Resources

About