Fill the GAP: Exploiting BERT for Pronoun Resolution

Abstract

In this paper, we describe our entry in the gendered pronoun resolution competition which achieved fourth place without data augmentation. Our method is an ensemble system of BERTs which resolves co-reference in an interaction space. We report four insights from our work: BERT's representations involve significant redundancy; modeling interaction effects similar to natural language inference models is useful for this task; there is an optimal BERT layer to extract representations for pronoun resolution; and the difference between the attention weights from the pronoun to the candidate entities was highly correlated with the correct label, with interesting implications for future work.

From Text Representation to Financial Market Prediction: A Literature Review

“…Different methods have adopted different data types. Some works on predicting stock markets only use text information from daily news [19,39,61,99,101], while others [18,25,34,124] consider a combination of textual and market-based features. Traditional models split data into train and test in chronological order for model training, while in the recursive model, training in rolling window strategy is used.…”

“…Liu, in [35], used long-term and short-term event embedding methods which contain the stack ELMO embedding of the t-days set of news headlines for the prediction of the S&P500 index. The authors of [21,25,35] used BERT-contextualized word embedding representation for market prediction. Farimani et al [16] proposed contextaware conceptual document representation to model the relevance between the news based on all the information in financial news titles and bodies via the clustering of contextualized BERT word embedding.…”

“…In this survey, we study methods for identifying the contextual information published in social media related to financial markets. Text mining techniques, such as a sentiment analysis [10][11][12][13], part of speech tagging (POS) [14,15], text representation, such as transformer-based word embedding [16][17][18][19][20][21][22], and machine learning techniques [23][24][25][26][27][28][29][30][31], have been used in this area after 2006. Recently, researchers have focused on using deep learning-based natural language processing (NLP), such as Bidirectional Encoder Representations from Transformer (BERT) [18,21,[32][33][34] or seq2seq architecture with an attention mechanism [20,[35][36][37][38], to structure textual web data.…”

“…Recently, researchers have focused on using deep learning-based natural language processing (NLP), such as Bidirectional Encoder Representations from Transformer (BERT) [18,21,[32][33][34] or seq2seq architecture with an attention mechanism [20,[35][36][37][38], to structure textual web data. BERT-contextualized word embedding, announced by Google in 2018, is used as a word sense disambiguation technique for summarizing and selecting important news for investors' behavior analysis [21,25]. Recent works have investigated the use of an attention mechanism in a deep encoder-decoder network that applies transfer learning from pre-trained models or calculating the impact of each news feed as well as assigning heavy weights for important time intervals in a time series [19,39,40].…”

Logit-based stock prediction network

“…The stock market forecast has always attracted many researchers and investors as a hot issue, but the stock market is a highly complex system, and the stock trend is affected by many factors such as technology, news, and public opinion [1] . The high volatility and uncertainty of the stock market make forecasting stocks difficult.…”