PropBank Comes of Age—Larger, Smarter, and more Diverse

Pradhan, S. Sandeep; Bonn, Julia; Myers, Skatje; Conger, Kathryn; O’gorman, Tim; Gung, James; Wright-bettner, Kristin; Palmer, Martha

doi:10.18653/v1/2022.starsem-1.24

Cited by 5 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To begin with, what is the predicate sense number? Similar to word senses in Wordnet (Fellbaum, 2010), the predicate senses in PropBank inside a predicate frame file are generally ordered from most to least frequently used, with the most common sense numbered 01 (Pradhan et al, 2022). The sense numbers (01, 02, 03, ... ) do not have any associated semantic meaning, and merely convey that one particular meaning of the predicate is more common than another.…”

Section: Predicate Sense Disambiguation Evaluationmentioning

confidence: 99%

PriMeSRL-Eval: A Practical Quality Metric for Semantic Role Labeling Systems Evaluation

Jindal,

Rademaker,

Tran

et al. 2023

Findings of the Association for Computational Linguistics: EACL 2023

View full text Add to dashboard Cite

Semantic role labeling (SRL) identifies predicate-argument structures in a sentence. This task is usually accomplished in four steps: predicate identification, predicate sense disambiguation, argument identification, and argument classification. Errors introduced at one step propagate to later steps. Unfortunately, the existing SRL evaluation scripts do not consider the full effect of this error propagation aspect. They either evaluate arguments independent of predicate sense (CoNLL09) or do not evaluate predicate sense at all (CoNLL05), yielding an inaccurate SRL model performance on the argument classification task. In this paper, we address key practical issues with existing evaluation scripts and propose a more strict SRL evaluation metric, PriMeSRL. We observe that by employing PriMeSRL, the quality evaluation of all SoTA SRL models drops significantly, and their relative rankings also change. We also show that PriMeSRL successfully penalizes actual failures in SoTA SRL models.

show abstract

Section: Predicate Sense Disambiguation Evaluationmentioning

confidence: 99%

PriMeSRL-Eval: A Practical Quality Metric for Semantic Role Labeling Systems Evaluation

Jindal,

Rademaker,

Tran

et al. 2023

Findings of the Association for Computational Linguistics: EACL 2023

View full text Add to dashboard Cite

show abstract

“…There is a much larger amount of PropBank-only data (e.g., Pradhan et al, 2013Pradhan et al, , 2022 than jointly labeled data. Inferring VerbNet labels from observed PropBank labels, therefore, is a realistic use case.…”

Section: Verbnet Completionmentioning

confidence: 99%

“…However, this raises the question of how to involve such inference during learning for better data efficiency. Furthermore, given the wider availability of PropBankonly data (e.g., Pradhan et al, 2013Pradhan et al, , 2022, how to efficiently benefit from such data also remains a question.…”

Section: Introductionmentioning

confidence: 99%

Learning Semantic Role Labeling from Compatible Label Sequences

Li,

Kazeminejad,

Brown

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Semantic role labeling (SRL) has multiple disjoint label sets, e.g., VerbNet and PropBank. Creating these datasets is challenging, therefore a natural question is how to use each one to help the other. Prior work has shown that cross-task interaction helps, but only explored multitask learning so far. A common issue with multitask setup is that argument sequences are still separately decoded, running the risk of generating structurally inconsistent label sequences (as per lexicons like SEMLINK). In this paper, we eliminate such issue with a framework that jointly models VerbNet and PropBank labels as one sequence. In this setup, we show that enforcing SEMLINK constraints during decoding constantly improves the overall F1. With special input constructions, our joint model infers VerbNet arguments from given PropBank arguments with over 99 F1. For learning, we propose a constrained marginal model that learns with knowledge defined in SEMLINK to further benefit from the large amounts of PropBankonly data. On the joint benchmark based on CoNLL05, our models achieve state-of-the-art F1's, outperforming the prior best in-domain model by 3.5 (VerbNet) and 0.8 (PropBank). For out-of-domain generalization, our models surpass the prior best by 3.4 (VerbNet) and 0.2 (PropBank).

show abstract

“…In an AMR graph, predicates and their corresponding arguments are represented by nodes. In this example, the go-02 predicate 2 (Palmer et al, 2005;Pradhan et al, 2022) has one argument, which is boy. AMR specifies the role of each argument with labeled edges.…”

Section: Introductionmentioning

confidence: 99%

CAMRA: Copilot for AMR Annotation

Cai,

Ahmed,

Bonn

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations

View full text Add to dashboard Cite

In this paper, we introduce CAMRA (Copilot for AMR Annotatations), a cutting-edge webbased tool designed for constructing Abstract Meaning Representation (AMR) from natural language text. CAMRA offers a novel approach to deep lexical semantics annotation such as AMR, treating AMR annotation akin to coding in programming languages. Leveraging the familiarity of programming paradigms, CAMRA encompasses all essential features of existing AMR editors, including example lookup, while going a step further by integrating Propbank roleset lookup as an autocomplete feature within the tool. Notably, CAMRA incorporates AMR parser models as coding copilots, greatly enhancing the efficiency and accuracy of AMR annotators. To demonstrate the tool's capabilities, we provide a live demo accessible at: https://camra.colorado.edu 1

show abstract

PropBank Comes of Age—Larger, Smarter, and more Diverse

Cited by 5 publications

References 15 publications

PriMeSRL-Eval: A Practical Quality Metric for Semantic Role Labeling Systems Evaluation

PriMeSRL-Eval: A Practical Quality Metric for Semantic Role Labeling Systems Evaluation

Learning Semantic Role Labeling from Compatible Label Sequences

CAMRA: Copilot for AMR Annotation

Contact Info

Product

Resources

About