A Span-based Dynamic Local Attention Model for Sequential Sentence Classification

Abstract

Sequential sentence classification aims to classify each sentence in the document based on the context in which sentences appear. Most existing work addresses this problem using a hierarchical sequence labeling network. However, they ignore considering the latent segment structure of the document, in which contiguous sentences often have coherent semantics. In this paper, we proposed a span-based dynamic local attention model that could explicitly capture the structural information by the proposed supervised dynamic local attention. We further introduce an auxiliary task called span-based classification to explore the span-level representations. Extensive experiments show that our model achieves better or competitive performance against state-of-theart baselines on two benchmark datasets.

Cross-domain multi-task learning for sequential sentence classification in research papers

“…Approaches for Abstracts: Deep learning has been the preferred approach for sentence classification in abstracts in recent years [15,19,28,34,65,75]. These approaches follow a common hierarchical sequence labelling architecture: (1) a word embedding layer encodes tokens of a sentence to word embeddings, (2) a sentence encoder transforms the word embeddings of a sentence to a sentence representation, (3) a context enrichment layer enriches all sentence representations of the abstract with context from surrounding sentences, and (4) an output layer predicts the label sequence.…”

“…Global Vectors (GloVe) [55], Word2Vec [47], or SciBERT [6] that is BERT [20] pre-trained on scientific text. For sentence encoding, a bidirectional long short-term memory (Bi-LSTM) [31] or a convolutional neural network (CNN) with various pooling strategies are utilised, while Yamada et al [75] and Shang et al [65] use the classification token ([CLS]) of BERT or SciBERT. To enrich sentences with further context, a recurrent neural network such as a Bi-LSTM or bidirectional gated recurrent unit (Bi-GRU) [13] is used.…”

“…To enrich sentences with further context, a recurrent neural network such as a Bi-LSTM or bidirectional gated recurrent unit (Bi-GRU) [13] is used. Shang et al [65] additionally exploit an attention-mechanism across sentences; however, it introduces quadratic runtime complexity that depends on the number of sentences. A conditional random field (CRF) [40] is mostly used as an output layer to capture the interdependence between classes.…”

“…Several approaches have been proposed for sequential sentence classification (e.g. [2,34,65]), and several datasets were annotated for various scientific domains (e.g. [18,23,28,67]).…”

“…The general paper text, however, exhibits longer passages without change of the semantic sentence class. Typically, deep learning is used for abstracts [15,19,28,34,65,75] since presumably more training data are available, whereas for full papers, also called zone identification, hand-crafted features and linear models have been suggested [2,4,23,44]. However, deep learning approaches have also been applied successfully to full papers in related tasks such as argumentation mining [41], document summarisation [1,16,21,27], or n-ary relation extraction [25,33,36].…”