Multi-Task Learning for Coherence Modeling

Farag, Youmna; Yannakoudakis, Helen

doi:10.18653/v1/p19-1060

Cited by 19 publications

(18 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MTL with BERT embeddings (MTL bert ): We replicate the previous MTL model but now use BERT embeddings (Devlin et al, 2019) to initialize the input words. Single-task learning (STL, Farag and Yannakoudakis, 2019): This model has the same architecture as MTL but only performs the coherence prediction task, excluding the grammatical role auxiliary objective. STL with BERT (STL bert ): This is the same as STL but uses BERT embeddings.…”

Section: Neural Coherence Modelsmentioning

confidence: 99%

“…Some models utilize structured representations of text (e.g. Egrid representations, Tien Nguyen and Joty, 2017;Joty et al, 2018) and others operate on unstructured text, taking advantage of neural models' ability to learn useful representations for the task (Li and Jurafsky, 2017;Logeswaran et al, 2018;Farag and Yannakoudakis, 2019;Moon et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

“…Coherence has typically been assessed by a model's ability to rank a well-organized document higher than its noisy counterparts created by corrupting sentence order in the original document (binary discrimination task), and neural models have achieved remarkable accuracy on this task. Recent efforts have targeted additional tasks such as recovering the correct sentence order (Logeswaran et al, 2018;Cui et al, 2018), evaluating on realistic data (Lai and Tetreault, 2018;Farag and Yannakoudakis, 2019) and focusing on open-domain models of coherence (Li and Jurafsky, 2017;. However, less attention has been directed to investigating and analyzing the properties of coherence that current models can capture, nor what knowledge is encoded in their representations and how it might relate to aspects of coherence.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Analyzing Neural Discourse Coherence Models

Farag¹,

Valvoda²,

Yannakoudakis³

et al. 2020

Proceedings of the First Workshop on Computational Approaches to Discourse

Self Cite

View full text Add to dashboard Cite

In this work, we systematically investigate how well current models of coherence can capture aspects of text implicated in discourse organisation. We devise two datasets of various linguistic alterations that undermine coherence and test model sensitivity to changes in syntax and semantics. We furthermore probe discourse embedding space and examine the knowledge that is encoded in representations of coherence. We hope this study shall provide further insight into how to frame the task and improve models of coherence assessment further. Finally, we make our datasets publicly available as a resource for researchers to use to test discourse coherence models.

show abstract

Section: Neural Coherence Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Analyzing Neural Discourse Coherence Models

Farag¹,

Valvoda²,

Yannakoudakis³

et al. 2020

Proceedings of the First Workshop on Computational Approaches to Discourse

Self Cite

View full text Add to dashboard Cite

show abstract

“…Coherence modeling is one of the essential aspects of natural language processing Mesgar et al, 2019;Moon et al, 2019;Farag and Yannakoudakis, 2019). A coherent text can facilitate understanding and avoid the confusion for reading comprehension.…”

Section: Introductionmentioning

confidence: 99%

BERT-enhanced Relational Sentence Ordering Network

Cui¹,

Li²,

Zhang³

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

In this paper, we introduce a novel BERTenhanced Relational Sentence Ordering Network (referred to as BERSON) by leveraging BERT for capturing a better dependency relationship among sentences to enhance the coherence modeling for the entire paragraph. In particular, we develop a new Relational Pointer Decoder (referred as RPD) by incorporating the relative ordering information into the pointer network with a Deep Relational Module (referred as DRM), which utilizes BERT to exploit the deep semantic connection and relative ordering between sentences. This enables us to strengthen both local and global dependencies among sentences. Extensive evaluations are conducted on six public datasets. The experimental results demonstrate the effectiveness and promise of BERSON, showing a significant improvement over the state-of-the-art by a wide margin. * Corresponding author An unordered set of sentences Coherent paragraph 1 Dan was walking during the night.1 Dan was walking during the night. 3 They tried to steal his book bag.2 A group of thieves surrounded him. 4 A bystander noticed them.3 They tried to steal his book bag. 2 A group of thieves surrounded him. 4 A bystander noticed them. 5 But she continued to walk away.5 But she continued to walk away.

show abstract

“…Despite some successes, techniques explored so far mainly rely on word sequence within a sentence. Farag and Yannakoudakis (2019) attempted to encode information about the types of grammatical roles, such as clausal modifiers of nouns and coordinating conjunction in a sentence obtained by using the Stanford Dependency Parser (Chen and Manning, 2014). for coherence modeling.…”

Section: Introductionmentioning

confidence: 99%

A Neural Local Coherence Analysis Model for Clarity Text Scoring

Muangkammuen¹,

Xu²,

Fukumoto³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Local coherence relation between two phrases/sentences, such as cause-effect, and contrast gives a strong influence on whether a text is well-structured or not. This paper follows the assumption and presents a method for scoring text clarity by utilizing local coherence between adjacent sentences. We hypothesize that the coherence knowledge learned from the different domain data is beneficial for capturing a well-structured text and thus helpful for scoring text clarity. We propose a text clarity scoring method that utilizes local coherence analysis with an out-domain setting, i.e., the training data for the source and target tasks are different from each other. The method based on the pre-trained language model BERT firstly trains the local coherence model as an auxiliary manner and then re-trains it together with the text clarity scoring model. The experimental results by using the PeerRead benchmark dataset show the improvement compared with a single model, scoring text clarity model 1 .

show abstract

Multi-Task Learning for Coherence Modeling

Cited by 19 publications

References 26 publications

Analyzing Neural Discourse Coherence Models

Analyzing Neural Discourse Coherence Models

BERT-enhanced Relational Sentence Ordering Network

A Neural Local Coherence Analysis Model for Clarity Text Scoring

Contact Info

Product

Resources

About