Pretrained Language Models for Sequential Sentence Classification

Cohan, Arman; Beltagy, Iz; King, Daniel; Dalvi, Bhavana; Weld, Daniel S.

doi:10.18653/v1/d19-1383

Cited by 90 publications

(95 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Of these, only one specifically performs sequential sentence classification (i.e. the task of providing labels for each of the sentences in the multi-sentence input) (Cohan et al, 2019). There exist non-PLM approaches to sequential sentence classification as well.…”

Section: Related Workmentioning

confidence: 99%

“…To involve direct textual context, we use a Windowed Sequential Sentence Classification method (WinSSC). Like Cohan et al (2019)'s method of using pre-trained language models for sequential sentence classification, WinSSC takes multiple sentences as its input sequence, generates embeddings for the separator tokens in the sequence, and classifies these embeddings with a linear layer that outputs as many labels as there are sentences in the input sequence. Prior to embedding, sequences are book-ended with the last sentence from the previous sequence, and the first sentence of the next sequence.…”

Section: Approachesmentioning

confidence: 99%

See 1 more Smart Citation

Context in Informational Bias Detection

Berg¹,

Markert

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Informational bias is bias conveyed through sentences or clauses that provide tangential, speculative or background information that can sway readers' opinions towards entities. By nature, informational bias is context-dependent, but previous work on informational bias detection has not explored the role of context beyond the sentence. In this paper, we explore four kinds of context for informational bias in English news articles: neighboring sentences, the full article, articles on the same event from other news publishers, and articles from the same domain (but potentially different events). We find that integrating event context improves classification performance over a very strong baseline. In addition, we perform the first error analysis of models on this task. We find that the best-performing context-inclusive model outperforms the baseline on longer sentences, and sentences from politically centrist articles.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Approachesmentioning

confidence: 99%

Context in Informational Bias Detection

Berg¹,

Markert

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…The above summary generated by standard copy mechanism miss some importance words, such as "obama" and "nominees". on many NLP tasks, including machine translation (Vaswani et al, 2017;Dehghani et al, 2019), sentence classification (Devlin et al, 2019;Cohan et al, 2019), and text summarization (Song et al, 2019;.…”

Section: Introductionmentioning

confidence: 99%

Self-Attention Guided Copy Mechanism for Abstractive Summarization

Xu¹,

Li²,

Yuan³

et al. 2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Copy module has been widely equipped in the recent abstractive summarization models, which facilitates the decoder to extract words from the source into the summary. Generally, the encoder-decoder attention is served as the copy distribution, while how to guarantee that important words in the source are copied remains a challenge. In this work, we propose a Transformer-based model to enhance the copy mechanism. Specifically, we identify the importance of each source word based on the degree centrality with a directed graph built by the self-attention layer in the Transformer. We use the centrality of each source word to guide the copy process explicitly. Experimental results show that the self-attention graph provides useful guidance for the copy distribution. Our proposed models significantly outperform the baseline methods on the CNN/Daily Mail dataset and the Gigaword dataset.

show abstract

“…Sun et al (2019a) proposed to construct an auxiliary sentence to solve aspect-based sentiment classification tasks. Cohan et al (2019) added extra separate tokens to obtain representations of each sentence to solve sequential sentence classification tasks. Sun et al (2019b) summarized several fine-tuning methods, including fusing text representations from different layers, utilizing multi-task learning, etc.…”

Section: Introductionmentioning

confidence: 99%

Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking

Yang¹,

Cao²,

Wen³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Automated Essay Scoring (AES) is a critical text regression task that automatically assigns scores to essays based on their writing quality. Recently, the performance of sentence prediction tasks has been largely improved by using Pre-trained Language Models via fusing representations from different layers, constructing an auxiliary sentence, using multitask learning, etc. However, to solve the AES task, previous works utilize shallow neural networks to learn essay representations and constrain calculated scores with regression loss or ranking loss, respectively. Since shallow neural networks trained on limited samples show poor performance to capture deep semantic of texts. And without an accurate scoring function, ranking loss and regression loss measures two different aspects of the calculated scores. To improve AES's performance, we find a new way to fine-tune pre-trained language models with multiple losses of the same task. In this paper, we propose to utilize a pretrained language model to learn text representations first. With scores calculated from the representations, mean square error loss and the batch-wise ListNet loss with dynamic weights constrain the scores simultaneously. We utilize Quadratic Weighted Kappa to evaluate our model on the Automated Student Assessment Prize dataset. Our model outperforms not only state-of-the-art neural models near 3 percent but also the latest statistic model. Especially on the two narrative prompts, our model performs much better than all other state-of-theart models.

show abstract

Pretrained Language Models for Sequential Sentence Classification

Cited by 90 publications

References 17 publications

Context in Informational Bias Detection

Context in Informational Bias Detection

Self-Attention Guided Copy Mechanism for Abstractive Summarization

Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking

Contact Info

Product

Resources

About