End-to-End Transformer-Based Models in Textual-Based NLP

Rahali, Abir; Akhloufi, Moulay A.

doi:10.3390/ai4010004

Cited by 51 publications

(20 citation statements)

References 238 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In practical applications, encoding and decoding 3 models are commonly used for processing. The encoder encodes the input text into a vector representation, and the decoder converts this vector into an output title 4 , which show in Figure 1.…”

Section: Overview Of Text Title Generation and Abstract Extraction Mo...mentioning

confidence: 99%

Research and implementation of title generation for news text

Li,

Tang,

Lin

2023

Fifth International Conference on Artificial Intelligence and Computer Science (AICS 2023)

View full text Add to dashboard Cite

This paper explores the natural language processing technology of news text, especially the algorithm and application of title generation and abstraction. Based on the analysis of the existing principle of automatic title generation and abstraction, the data sets are preprocessed by text cleaning and word quantization according to the objective and accurate features of news text. Then, the Roberta, GPT-2, and T5-pegasus models were evaluated, and the method of model fusion and integration based on the principle of accuracy priority was proposed to improve the accuracy of title generation by 3%. Finally, the news text processing system is constructed based on the optimized model. Experimental results show that the proposed algorithm has high reliability and robustness. This topic has been applied, and the application shows that the results of the news text processing system are accurate and objective, and have high adaptability.

show abstract

Section: Overview Of Text Title Generation and Abstract Extraction Mo...mentioning

confidence: 99%

Research and implementation of title generation for news text

Li,

Tang,

Lin

2023

Fifth International Conference on Artificial Intelligence and Computer Science (AICS 2023)

View full text Add to dashboard Cite

show abstract

“…В контексте обнаружения ВПО для этого используются такие методы, как Word2Vec [10], HMM2Vec [11], BERT [12] и ELMo [13]. BERT показал наибольшую эффективность в системах MalBERT [14] и её усовершенствованной версии MalBERTv2 [15]. Однако, в указанных выше системах используется подход статического анализа.…”

Section: Introductionunclassified

A Method for Detecting Ransomware Based on the Analysis of the Behavioral Report of the Executable Object

2024

Cybersecurity Issues

View full text Add to dashboard Cite

Цель работы: разработка метода обнаружения программ-вымогателей на основе анализа последовательностей API-вызовов и системных вызовов.Метод исследования: анализ записей в поведенческом отчёте продукта виртуализации с использованием алгоритма глубокого обучения DeBERTa-V3.Полученный результат: несмотря на большое разнообразие семейств и вариаций семейств программвымогателей, используемых злоумышленниками в компьютерных атаках, все они оставляют следы своей работы в атакуемой инфраструктуре. Одним из способов выявления вредоносного программного обеспечения и предотвращения заражения является использование технологии «Песочница», в том числе для выявления скрытых возможностей исследуемого объекта и аномалий его поведения. Функционирование любой компьютерной программы можно представить в виде набора записей его действий в отчёте поведения, которые можно рассматривать в качестве признаков объекта. В работе проведен анализ отчётов поведения программ-вымогателей. На сформированном наборе данных с использованием алгоритма глубокого обучения построена модель, позволяющая в дальнейшем выявлять вредоносные объекты, а также описан метод обнаружения программ-вымогателей.Практическая ценность состоит в создании метода обнаружения программ-вымогателей на основе анализа поведенческого отчета исполняемого объекта с использованием алгоритма глубокого обучения DeBERTa-V3.

show abstract

“…Transformers, a neural network architecture, have garnered considerable interest in both Natural Language Processing (NLP) and time series analysis. Their capacity to manage long-range dependencies and parallel processing has propelled their popularity in these domains (Rahali and Akhloufi, 2023). Utilizing self-attention or scaled dot-product attention, Transformers can assess the significance of inputs within a sequence, enabling the capture of intricate data patterns.…”

Section: ) Introductionmentioning

confidence: 99%

Enhancing Pneumonia Detection using Vision Transformer with Dynamic Mapping Re-Attention Mechanism

Labbaf Khaniki,

Mirzaeibonehkhater,

Manthouri

2023

2023 13th International Conference on Computer and Knowledge Engineering (ICCKE)

View full text Add to dashboard Cite

Fault detection and diagnosis (FDD) is a crucial task for ensuring the safety and efficiency of industrial processes. We propose a novel FDD methodology for the Tennessee Eastman Process (TEP), a widely used benchmark for chemical process control. The model employs two separate Transformer branches, enabling independent processing of input data and potential extraction of diverse information. A novel attention mechanism, Gated Dynamic Learnable Attention (GDLAttention), is introduced which integrates a gating mechanism and dynamic learning capabilities. The gating mechanism modulates the attention weights, allowing the model to focus on the most relevant parts of the input. The dynamic learning approach adapts the attention strategy during training, potentially leading to improved performance. The attention mechanism uses a bilinear similarity function, providing greater flexibility in capturing complex relationships between query and key vectors. In order to assess the effectiveness of our approach, we tested it against 21 and 18 distinct fault scenarios in TEP, and compared its performance with several established FDD techniques. The outcomes indicate that the method outperforms others in terms of accuracy, false alarm rate, and misclassification rate. This underscores the robustness and efficacy of the approach for FDD in intricate industrial processes.

show abstract

End-to-End Transformer-Based Models in Textual-Based NLP

Cited by 51 publications

References 238 publications

Research and implementation of title generation for news text

Research and implementation of title generation for news text

A Method for Detecting Ransomware Based on the Analysis of the Behavioral Report of the Executable Object

Enhancing Pneumonia Detection using Vision Transformer with Dynamic Mapping Re-Attention Mechanism

Contact Info

Product

Resources

About