Multi-Task Learning for Argumentation Mining in Low-Resource Settings

Schulz, Cláudia; Eger, Steffen; Daxenberger, Johannes; Kahse, Tobias; Gurevych, Iryna

doi:10.18653/v1/n18-2006

Cited by 56 publications

(80 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Related to the study of diagnostic reasoning is argumentative reasoning, which has recently received growing attention from the NLP community. The focus has been on identifying argument components (Lippi and Torroni 2015;Schulz et al 2018) or whole arguments, made of components (such as premises and claims) as well as attacking and supporting relations between them (Menini et al 2018;Habernal and Gurevych 2017). Like us, Stab and Gurevych (2014) and Nguyen and Litman (2018) investigate arguments in an educational setting by automatically identifying arguments in students' persuasive essays.…”

Section: Related Workmentioning

confidence: 99%

“…Challenges C1 and C2 imply that we are dealing with a multiclass sequence labeling task, as for example encountered in the related task of argument component identification (Schulz et al 2018). This is commonly modeled by assigning a label to each token that expresses both the type of segment, here the type of epistemic activity A = {HG, EG, EE, DC}, and the segment boundaries in terms of BIO-labels S = {B, I, O}, indicating the beginning (B), continuation (I), or absence (O) of a segment.…”

Section: Modeling the Taskmentioning

confidence: 99%

See 1 more Smart Citation

Challenges in the Automatic Analysis of Students’ Diagnostic Reasoning

Schulz

Meyer

Gurevych

2019

AAAI

Self Cite

View full text Add to dashboard Cite

Diagnostic reasoning is a key component of many professions. To improve students' diagnostic reasoning skills, educational psychologists analyse and give feedback on epistemic activities used by these students while diagnosing, in particular, hypothesis generation, evidence generation, evidence evaluation, and drawing conclusions. However, this manual analysis is highly time-consuming. We aim to enable the large-scale adoption of diagnostic reasoning analysis and feedback by automating the epistemic activity identification. We create the first corpus for this task, comprising diagnostic reasoning selfexplanations of students from two domains annotated with epistemic activities. Based on insights from the corpus creation and the task's characteristics, we discuss three challenges for the automatic identification of epistemic activities using AI methods: the correct identification of epistemic activity spans, the reliable distinction of similar epistemic activities, and the detection of overlapping epistemic activities. We propose a separate performance metric for each challenge and thus provide an evaluation framework for future research. Indeed, our evaluation of various state-of-the-art recurrent neural network architectures reveals that current techniques fail to address some of these challenges.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Modeling the Taskmentioning

confidence: 99%

Challenges in the Automatic Analysis of Students’ Diagnostic Reasoning

Schulz

Meyer

Gurevych

2019

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…Users in social media platforms usually express emotions or quick messages with very little argumentation, however the introduction of argumentative features can enhance other NLP tasks [60,66]. Both micro [93] and macro [94] analysis have the attention of the research community, whereas they have been approaches that combine them [87,88]. Another research topic that has gained the interest of the research community is the reconstruction of implicit warrants, although the existing research papers [8,9,29] do not utilize social media as source.…”

Section: Relations Identificationmentioning

confidence: 99%

“…Deep learning techniques are able to handle a great volume of data in an unsupervised or semi-supervised way and they have achieved break-trough results in NLP field. Deep learning has been applied [93,39,9] in AM, but does not seem to overpass other ML algorithms, mainly because of the limited available datasets, however more research should take place in order safe conclusion to be drawn.…”

Section: Future Directions: Semi-supervision and Background Knowledgementioning

confidence: 99%

The evolution of argumentation mining: From models to social media and emerging tools

Lytos

Λάγκας²,

Sarigiannidis

et al. 2019

Information Processing & Management

View full text Add to dashboard Cite

Argumentation mining is a rising subject in the computational linguistics domain focusing on extracting structured arguments from natural text, often from unstructured or noisy text. The initial approaches on modeling arguments was aiming to identify a flawless argument on specific fields (Law, Scientific Papers) serving specific needs (completeness, effectiveness). With the emerge of Web 2.0 and the explosion in the use of social media both the diffusion of the data and the argument structure have changed. In this survey article, we bridge the gap between theoretical approaches of argumentation mining and pragmatic schemes that satisfy the needs of social media generated data, recognizing the need for adapting more flexible and expandable schemes, capable to adjust to the argumentation conditions that exist in social media. We review, compare, and classify existing approaches, techniques and tools, identifying the positive outcome of combining tasks and features, and eventually propose a conceptual architecture framework. The proposed theoretical framework is an argumentation mining scheme able to identify the distinct sub-tasks and capture the needs of social media text, revealing the need for adopting more flexible and extensible frameworks.

show abstract

“…However, since T is finite, overfitting the training set might lead to poor generalization performance. One way to avoid fitting Equation 1 too # train # dev ES Bollmann et al (2018) 5k 12k-46k Yes 400-700 100-200 Yes Makarov and Clematide (2018) 100 1k Yes Sharma et al (2018) 100 100 Yes Schulz et al (2018) 1k-21k 9k N/A Upadhyay et al (2018) 500 1k Yes closely is early stopping: a separate development or validation set is used to end training as soon as the loss on the development set L D (θ) starts increasing or model performance on the development set D starts decreasing. The best set of parameters θ is used in the final model.…”

Section: Introductionmentioning

confidence: 99%

Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set

Kann¹,

Cho²,

Bowman³

2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

Development sets are impractical to obtain for real low-resource languages, since using all available data for training is often more effective. However, development sets are widely used in research papers that purport to deal with low-resource natural language processing (NLP). Here, we aim to answer the following questions: Does using a development set for early stopping in the low-resource setting influence results as compared to a more realistic alternative, where the number of training epochs is tuned on development languages? And does it lead to overestimation or underestimation of performance? We repeat multiple experiments from recent work on neural models for low-resource NLP and compare results for models obtained by training with and without development sets. On average over languages, absolute accuracy differs by up to 1.4%. However, for some languages and tasks, differences are as big as 18.0% accuracy. Our results highlight the importance of realistic experimental setups in the publication of lowresource NLP research results.

show abstract

Multi-Task Learning for Argumentation Mining in Low-Resource Settings

Cited by 56 publications

References 24 publications

Challenges in the Automatic Analysis of Students’ Diagnostic Reasoning

Challenges in the Automatic Analysis of Students’ Diagnostic Reasoning

The evolution of argumentation mining: From models to social media and emerging tools

Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set

Contact Info

Product

Resources

About