Contrastive Attention Mechanism for Abstractive Sentence Summarization

Duan, Xiangyu; Yu, Hoongfei; Yin, Mingming; Zhang, Min; Luo, Weihua; Zhang, Yue

doi:10.48550/arxiv.1910.13114

Cited by 3 publications

(2 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This method extends the influence of source text keywords on generating the final summary. Duan [23] added a comparison attention mechanism to the Transformer. This attention is calculated from the parameters and weights inside the Transformer to increase attention to irrelevant information between the target language reference summary and the source language text.…”

Section: Auxiliary Generation Methodsmentioning

confidence: 99%

A Multitask Cross-Lingual Summary Method Based on ABO Mechanism

Wan

Zhao

2023

Applied Sciences

View full text Add to dashboard Cite

Recent cross-lingual summarization research has pursued the use of a unified end-to-end model which has demonstrated a certain level of improvement in performance and effectiveness, but this approach stitches together multiple tasks and makes the computation more complex. Less work has focused on alignment relationships across languages, which has led to persistent problems of summary misordering and loss of key information. For this reason, we first simplify the multitasking by converting the translation task into an equal proportion of cross-lingual summary tasks so that the model can perform only cross-lingual summary tasks when generating cross-lingual summaries. In addition, we splice monolingual and cross-lingual summary sequences as an input so that the model can fully learn the core content of the corpus. Then, we propose a reinforced regularization method based on the model to improve its robustness, and build a targeted ABO mechanism to enhance the semantic relationship alignment and key information retention of the cross-lingual summaries. Ablation experiments are conducted on three datasets of different orders of magnitude to demonstrate the effective enhancement of the model by the optimization approach; they outperform the mainstream approaches on the cross-lingual summarization task and the monolingual summarization task for the full dataset. Finally, we validate the model’s capabilities on a cross-lingual summary dataset of professional domains, and the results demonstrate its superior performance and ability to improve cross-lingual sequencing.

show abstract

Section: Auxiliary Generation Methodsmentioning

confidence: 99%

A Multitask Cross-Lingual Summary Method Based on ABO Mechanism

Wan

Zhao

2023

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…The main issues of applying CL to the language field for the same purpose as in image settings are the difficulty of establishing a general strategy for text 'augmentation', as well as determining many negative samples of an anchor point (Rethmeier and Augenstein 2021). Despite having less popularity than the image counterpart (Jaiswal et al 2021) CL in NLP has been used to pre-train zero-shot predictions (Rethmeier and Augenstein 2020; Pappas and Henderson 2019), and also to improve specific tasks such as language modeling (Logeswaran and Lee 2018;Giorgi et al 2020), text summarization (Duan et al 2019), and many others (Rethmeier and Augenstein 2021). In the previous approaches, the core purpose of the contrastive learning is to obtain more accurate representations or embeddings of texts/sentences for task-specific purposes.…”

Section: Contrastive Learningmentioning

confidence: 99%

Adversarial Training with Contrastive Learning in NLP

Rim,

Heo,

Choi

2021

Preprint

View full text Add to dashboard Cite

For years, adversarial training has been extensively studied in natural language processing (NLP) settings. The main goal is to make models robust so that similar inputs derive in semantically similar outcomes, which is not a trivial problem since there is no objective measure of semantic similarity in language. Previous works use an external pre-trained NLP model to tackle this challenge, introducing an extra training stage with huge memory consumption during training. However, the recent popular approach of contrastive learning in language processing hints a convenient way of obtaining such similarity restrictions. The main advantage of the contrastive learning approach is that it aims for similar data points to be mapped close to each other and further from different ones in the representation space. In this work, we propose adversarial training with contrastive learning (ATCL) to adversarially train a language processing task using the benefits of contrastive learning. The core idea is to make linear perturbations in the embedding space of the input via fast gradient methods (FGM) and train the model to keep the original and perturbed representations close via contrastive learning.In NLP experiments, we applied ATCL to language modeling and neural machine translation tasks. The results show not only an improvement in the quantitative (perplexity and BLEU) scores when compared to the baselines, but ATCL also achieves good qualitative results in the semantic level for both tasks without using a pre-trained model.

show abstract

ModVAR: A Multimodal Framework for Comprehensive Driver Variant Prediction in Cancer

Wang,

Yang,

Chen

et al. 2024

Preprint

View full text Add to dashboard Cite

Cancer's genomic complexity poses significant challenges in identifying the crucial driver variants that fuel oncogenesis. To overcome these obstacles, we present ModVAR, an innovative multimodal model that seamlessly integrates DNA sequences, protein tertiary structures, and cancer omics data to predict driver variants with exceptional precision. Leveraging pre-trained models DNAbert2 and ESMFold, combined with a novel self-supervised strategy for omics data, ModVAR consistently outperforms 14 leading prediction methods. It achieves an outstanding AUROC of 0.985 in clinical benchmarks and 0.952 in experimental assay-related benchmarks, underscoring its effectiveness in identifying both clinically and experimentally validated variants. ModVAR also excels in predicting cancer indels, with an AUROC of 0.901, and demonstrates remarkable generalization with an AUROC of 0.981 for non-cancer disease variants. These results underscore ModVAR's robustness and versatility, positioning it as a powerful tool in genomics and precision medicine. Additionally, its application to COSMIC has generated a publicly available resource of 3,971,946 annotated variants, driving an advancement in cancer research and personalized therapy.

show abstract

Contrastive Attention Mechanism for Abstractive Sentence Summarization

Cited by 3 publications

References 19 publications

A Multitask Cross-Lingual Summary Method Based on ABO Mechanism

A Multitask Cross-Lingual Summary Method Based on ABO Mechanism

Adversarial Training with Contrastive Learning in NLP

ModVAR: A Multimodal Framework for Comprehensive Driver Variant Prediction in Cancer

Contact Info

Product

Resources

About