Long-Range Transformer Architectures for Document Understanding

Douzon, Thibault; Duffner, Stefan; García, Christophe; Espinas, Jérémy

doi:10.1007/978-3-031-41501-2_4

Cited by 4 publications

(2 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, refs. [20,21] introduces the self-attention approach, which reduces the number of sequential computations by short paths between distant words in entire paragraphs. It concludes that these short paths are particularly useful for learning strong semantic feature extractors.…”

Section: Related Workmentioning

confidence: 99%

Neural Machine Translation with CARU-Embedding Layer and CARU-Gated Attention Layer

Im,

Chan

2024

Mathematics

View full text Add to dashboard Cite

The attention mechanism performs well for the Neural Machine Translation (NMT) task, but heavily depends on the context vectors generated by the attention network to predict target words. This reliance raises the issue of long-term dependencies. Indeed, it is very common to combine predicates with postpositions in sentences, and the same predicate may have different meanings when combined with different postpositions. This usually poses an additional challenge to the NMT study. In this work, we observe that the embedding vectors of different target tokens can be classified by part-of-speech, thus we analyze the Natural Language Processing (NLP) related Content-Adaptive Recurrent Unit (CARU) unit and apply it to our attention model (CAAtt) and embedding layer (CAEmbed). By encoding the source sentence with the current decoded feature through the CARU, CAAtt is capable of achieving translation content-adaptive representations, which attention weights are contributed and enhanced by our proposed L1expNx normalization. Furthermore, CAEmbed aims to alleviate long-term dependencies in the target language through partial recurrent design, performing the feature extraction in a local perspective. Experiments on the WMT14, WMT17, and Multi30k translation tasks show that the proposed model achieves improvements in BLEU scores and enhancement of convergence over the attention-based plain NMT model. We also investigate the attention weights generated by the proposed approaches, which indicate that refinement over the different combinations of adposition can lead to different interpretations. Specifically, this work provides local attention to some specific phrases translated in our experiment. The results demonstrate that our approach is effective in improving performance and achieving a more reasonable attention distribution compared to the state-of-the-art models.

show abstract

Section: Related Workmentioning

confidence: 99%

Neural Machine Translation with CARU-Embedding Layer and CARU-Gated Attention Layer

Im,

Chan

2024

Mathematics

View full text Add to dashboard Cite

show abstract

“…Adversaries could exploit overfitting in LLMs to infer details about individual training examples, further complicating the privacy landscape [38,49]. The necessity for ongoing research and development in privacy-preserving methods was highlighted as a critical component in safeguarding the future deployment of LLMs [50,51].…”

Section: Privacy Breaches In Llm Modelsmentioning

confidence: 99%

Exploiting Privacy Vulnerabilities in Open Source LLMs Using Maliciously Crafted Prompts

Choquet,

Aizier,

Bernollin

2024

Preprint

View full text Add to dashboard Cite

The proliferation of AI technologies has brought to the forefront concerns regarding the privacy and security of user data, particularly with the increasing deployment of powerful language models such as Llama. A novel concept investigated involves inducing privacy breaches through maliciously crafted prompts, highlighting the potential for these models to inadvertently reveal sensitive information. The study systematically evaluated the vulnerabilities of the Llama model, employing an automated framework to test and analyze its responses to a variety of crafted inputs. Findings reveal significant security flaws, demonstrating the model's susceptibility to adversarial attacks that could compromise user privacy. Comprehensive analysis provided insights into the types of prompts most effective in eliciting private data, and the study demonstrates the necessity for robust regulatory frameworks and advanced security measures. The implications of these findings are profound, calling for immediate action to enhance the security protocols of LLMs and protect against potential privacy breaches. Enhanced regulatory oversight and continuous innovation in privacy-preserving techniques are crucial to ensuring the safe deployment of LLMs in various applications. The insights derived from this research contribute to a deeper understanding of LLM vulnerabilities and the urgent need for improved safeguards to prevent data leakage and unauthorized access.

show abstract

GraphKD: Exploring Knowledge Distillation Towards Document Object Detection with Structured Graph Creation

Banerjee,

Biswas,

Lladós

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Long-Range Transformer Architectures for Document Understanding

Cited by 4 publications

References 19 publications

Neural Machine Translation with CARU-Embedding Layer and CARU-Gated Attention Layer

Neural Machine Translation with CARU-Embedding Layer and CARU-Gated Attention Layer

Exploiting Privacy Vulnerabilities in Open Source LLMs Using Maliciously Crafted Prompts

GraphKD: Exploring Knowledge Distillation Towards Document Object Detection with Structured Graph Creation

Contact Info

Product

Resources

About