How compatible are our discourse annotation frameworks? Insights from mapping RST-DT and PDTB annotations

Abstract

Discourse-annotated corpora are an important resource for the community, but they are often annotated according to different frameworks. This makes joint usage of the annotations difficult, preventing researchers from searching the corpora in a unified way, or using all annotated data jointly to train computational systems. Several theoretical proposals have recently been made for mapping the relational labels of different frameworks to each other, but these proposals have so far not been validated against existing annotations. The two largest discourse relation annotated resources, the Penn Discourse Treebank and the Rhetorical Structure Theory Discourse Treebank, have however been annotated on the same texts, allowing for a direct comparison of the annotation layers. We propose a method for automatically aligning the discourse segments, and then evaluate existing mapping proposals by comparing the empirically observed against the proposed mappings. Our analysis highlights the influence of segmentation on subsequent discourse relation labelling, and shows that while agreement between frameworks is reasonable for explicit relations, agreement on implicit relations is low. We identify several sources of systematic discrepancies between the two annotation schemes and discuss consequences for future annotation and for usage of the existing resources.

Crowdsourcing Discourse Relation Annotations by a Two-Step Connective Insertion Task

“…The relation definitions of PDTB and RST-DT do not directly map with each other. In order to compare the annotations of both resources with the crowdsourced labels, we converted the RST labels to PDTB labels according to the Unifying Dimensions interlingua (Demberg et al, 2017). The results showed that the distributions of the crowd-sourced labels overlapped with both PDTB and RST-DT annotations, except for INSTANTIA-TIONS (see discussion).…”

Multi-lingual and Cross-genre Discourse Unit Segmentation

“…RST, SDRT or CCR) is a promising direction, but also one that needs more research. Earlier work in this direction ( (Demberg et al, 2017), (Sanders et al, 2018) and ) may help in the definition of a unifying minimal segment for future attempts at the segmentation task.…”

“…Iruskieta et al (2016) look at a particular kind of segment and detect central units in both Basque and Brazilian Portuguese, where they define central units (CUs) to be units that "(do) not function as satellite of any other unit or text span.". Earlier work on unifying discourse parsing frameworks is described in Rehbein et al (2016), Benamara and Taboada (2015), Bunt and Prasad (2016), Chiarcos (2014) and Sanders et al (2018) from a theoretical perspective, and in Demberg et al (2017) from a practical perspective, but their main focus is on relation senses. Although this presupposes some sort of mapping of units, language-and data-set individual segmentation can be, and in many cases is used.…”

Toward Cross-theory Discourse Relation Annotation

“…A notable exception is the practical approach based on the PDTB and the RST-DT described by Demberg et al (2017). The PDTB (Prasad et al, 2008) is annotated on the same set as the RST-DT (Carlson et al, 2002), but the former is considerably larger, with over 1.3m tokens compared to ca.…”