A Deep Learning Model Based on BERT and Sentence Transformer for Semantic Keyphrase Extraction on Big Social Data

Devika, R.; Subramaniyaswamy, V.; Mahenthar, C Sakthi Jay; Varadarajan, Vijayakumar; Kotecha, Ketan

doi:10.1109/access.2021.3133651

Cited by 34 publications

(15 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…e PIA attention mechanism is based on the simple idea that any natural language sentence can be transformed into a big integer in a Computational Intelligence and Neuroscience power-of-two number system using a simple polynomial transformation. By "power-of-two" here we mean that the radix x of the target number system is a power-of-two expressed as x � 2 n for some integer n. For example, the hexadecimal system is a power-of-two number system in which the radix x is 2 4 . e resulting big integer is then converted to a binary vector that encodes the coefficients of the transformation polynomial.…”

Section: E Pia Attention Mechanism: a Power-of-two Binary Polynomial ...mentioning

confidence: 99%

“…Transformers are currently the dominant approach in natural language processing (NLP) [ 1 , 2 ]. Since their introduction in 2017 [ 3 ], they have been successfully applied in various areas of NLP, including semantic key phrase extraction [ 4 ], hyperspectral image classification [ 5 ], multidimensional essay scoring [ 6 ], relation extraction [ 7 ], speech recognition [ 8 ], sentiment classification [ 9 ], geospatial market segmentation [ 10 ], fake news detection [ 11 ], question answering [ 12 ], text summarization [ 13 ], and text generation [ 14 ]. Good surveys of transformers and related attention technologies can be found in these references [ 45 , 46 ].…”

Section: Related Workmentioning

confidence: 99%

“…Transformers are currently state-of-the-art models in neural machine translation [1,2]. Since their introduction in 2017 [3], transformers have consistently produced state-of-the-art results in many areas of NLP and NMT [4][5][6][7][8][9][10][11][12][13][14]. However, based on recent results, it appears that transformers are reaching their limits.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Beyond the Transformer: A Novel Polynomial Inherent Attention (PIA) Model and Its Great Impact on Neural Machine Translation

ElAffendi

Alrajhi

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

This paper describes a novel polynomial inherent attention (PIA) model that outperforms all state-of-the-art transformer models on neural machine translation (NMT) by a wide margin. PIA is based on the simple idea that natural language sentences can be transformed into a special type of binary attention context vectors that accurately capture the semantic context and the relative dependencies between words in a sentence. The transformation is performed using a simple power-of-two polynomial transformation that maintains strict consistent positioning of words in the resulting vectors. It is shown how this transformation reduces the neural machine translation process to a simple neural polynomial regression model that provides excellent solutions to the alignment and positioning problems haunting transformer models. The test BELU scores obtained on the WMT-2014 data set are 75.07 BELU for the EN-FR data set and 66.35 BELU for the EN-DE data set—well above accuracies achieved by state-of-the-art transformer models for the same data sets. The improvements are, respectively, 65.7% and 87.42%.

show abstract

Section: E Pia Attention Mechanism: a Power-of-two Binary Polynomial ...mentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Beyond the Transformer: A Novel Polynomial Inherent Attention (PIA) Model and Its Great Impact on Neural Machine Translation

ElAffendi

Alrajhi

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…The Adjacent weighting matrix is calculated based on semantic knowledge and page knowledge. Since the semantic knowledge of advertising is mainly composed of sentences, here we use the output of the sentence transformer [7] as the semantic knowledge vectors s ∈ R k . As mentioned in section 2.1.3, we count the total number of ads placed on specific page channels and aggregate the average value of user interaction knowledge of historical ads placed on a specific page channel as Page knowledge vectors s ∈ R d .…”

Section: Graph Creationmentioning

confidence: 99%

Learning Graph Meta Embeddings for Cold-Start Ads in Click-Through Rate Prediction

Ouyang

Zhang

Ren

et al. 2021

Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Click-through rate (CTR) prediction is one of the most central tasks in online advertising systems. Recent deep learning-based models that exploit feature embedding and high-order data nonlinearity have shown dramatic successes in CTR prediction. However, these models work poorly on cold-start ads with new IDs, whose embeddings are not well learned yet. In this paper, we propose Graph Meta Embedding (GME) models that can rapidly learn how to generate desirable initial embeddings for new ad IDs based on graph neural networks and meta learning. Previous works address this problem from the new ad itself, but ignore possibly useful information contained in existing old ads. In contrast, GMEs simultaneously consider two information sources: the new ad and existing old ads. For the new ad, GMEs exploit its associated attributes. For existing old ads, GMEs first build a graph to connect them with new ads, and then adaptively distill useful information. We propose three specific GMEs from different perspectives to explore what kind of information to use and how to distill information. In particular, GME-P uses Pre-trained neighbor ID embeddings, GME-G uses Generated neighbor ID embeddings and GME-A uses neighbor Attributes. Experimental results on three real-world datasets show that GMEs can significantly improve the prediction performance in both cold-start (i.e., no training data is available) and warm-up (i.e., a small number of training samples are collected) scenarios over five major deep learning-based CTR prediction models. GMEs can be applied to conversion rate (CVR) prediction as well.

show abstract

“…Python was used to program the AI because it lets users easily express many complex and high-level tasks concisely, as well as offers a good platform for developing more specialized objects that are directly suited to scientific work (Perez et al, 2011) such as sentence transformer framework to compute semantic similarity and develop the language models (Devika et al, 2021) to answer the research question. Language models (such as BERT and MPNET) represent individual words with semantically fixed-length vectors that make possible natural language processing (NLP) (Greiner-Petter et al, 2020).…”

Section: Artificial Intelligence Training and Preparationmentioning

confidence: 99%

Comparing Two Artificial Intelligence Language Modeling to Evaluate Construction Schedule Understanding

Sulbaran

EPiC Series in Built Environment

View full text Add to dashboard Cite

Construction schedules are critical for construction projects. Creating a construction schedule is complex. Because it requires a keen ability to understand construction documents and methods. Furthermore, a single construction project could be completed following multiple construction schedules. The preparation of construction schedules is currently done by humans. However, with the advent of Artificial Intelligence (AI), it is possible to think of a future where humans could be assisted by an AI to develop construction schedules. An important step toward having an AI assisting humans in the creation of construction schedules is for the AI to understand construction schedules. Thus, this paper presents the comparison of two language modeling to evaluate an AI's ability to understand construction schedules. This comparison was done following a quantitative experimental research methodology where the independent variables were two AI language models (the Bidirectional Encoder Representation Transformers and the Masked and Permuted Pre-training for Language Understanding) and the dependent variables were accuracy, precision, recall, and F1 scores of the language modeling to understand construction schedule activities. The results demonstrate the impact of language models on the ability of the AI to understand construction schedules. The Masked and Permuted Pre-training for Language Understanding language model had an overall superior performance in understanding construction scheduling activities. This is important as supports the need to expand research projects as the one presented in his paper to identify the best language modeling and improve it for the construction industry.

show abstract

A Deep Learning Model Based on BERT and Sentence Transformer for Semantic Keyphrase Extraction on Big Social Data

Cited by 34 publications

References 25 publications

Beyond the Transformer: A Novel Polynomial Inherent Attention (PIA) Model and Its Great Impact on Neural Machine Translation

Beyond the Transformer: A Novel Polynomial Inherent Attention (PIA) Model and Its Great Impact on Neural Machine Translation

Learning Graph Meta Embeddings for Cold-Start Ads in Click-Through Rate Prediction

Comparing Two Artificial Intelligence Language Modeling to Evaluate Construction Schedule Understanding

Contact Info

Product

Resources

About