Implicit Discourse Relation Classification: We Need to Talk about Evaluation

Abstract

Implicit relation classification on Penn Discourse TreeBank (PDTB) 2.0 is a common benchmark task for evaluating the understanding of discourse relations. However, the lack of consistency in preprocessing and evaluation poses challenges to fair comparison of results in the literature. In this work, we highlight these inconsistencies and propose an improved evaluation protocol. Paired with this protocol, we report strong baseline results from pretrained sentence encoders, which set the new state-of-the-art for PDTB 2.0. Furthermore, this work is the first to explore fine-grained relation classification on PDTB 3.0. We expect our work to serve as a point of comparison for future work, and also as an initiative to discuss models of larger context and possible data augmentations for downstream transferability.

Not Just Classification: Recognizing Implicit Discourse Relation on Joint Modeling of Classification and Generation

“…Table 4 shows the performance of our model on PDTB-Ji at the top-level and second-level classes. Consistent with Kim et al (2020)'s conclusion, BERT is indeed the best baseline, achieves 51.88 and 36.10 in Micro-F1 and Macro-F1 at the secondlevel and 63.91 and 55.13 in Micro-F1 and Macro-F1 at the top-level, respectively.…”

“…PDTB: following previous work (Ji and Eisenstein, 2015;Kim et al, 2020), we adopt the mostused dataset splitting PDTB-Ji that takes the sections 2-20 as the training set, 0-1 as the development set, and 21-22 for testing.…”

Contextualized and Generalized Sentence Representations by Contrastive Self-Supervised Learning: A Case Study on Discourse Relation Analysis

“…We focus on implicit discourse relation analysis, where no explicit discourse marker exists. Following Kim et al (2020), we use the Level-2 labels with more than 100 examples and use 12-fold crossvalidation.…”

“…The evaluation protocol is 5-fold cross-validation. Following Kim et al (2020), each fold is split at the document level rather than the individual example level.…”

“…Unused BERTBASE (Kim et al, 2020) 57.60 XLNetBASE (Kim et al, 2020) 60 (Kim et al, 2020). Gen and Con indicate that the encoder is further pretrained by optimizing the generative objective and the contrastive objective, respectively.…”