The importance of fillers for text representations of speech transcripts

Dinkar, Tanvi; Colombo, Pierre; Labeau, Matthieu; Clavel, Chloé

doi:10.48550/arxiv.2009.11340

Cited by 3 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Emotion classification (Dinkar et al, 2020;Garcia et al, 2019;Colombo et al, 2021a;Witon et al, 2018) where score are continuous.…”

Section: B5 Possible Extensionsmentioning

confidence: 99%

What are the best systems? New perspectives on NLP Benchmarking

Colombo¹,

Noiry²,

Irurozki³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In Machine Learning, a benchmark refers to an ensemble of datasets associated with one or multiple metrics together with a way to aggregate different systems performances. They are instrumental in (i) assessing the progress of new methods along different axes and (ii) selecting the best systems for practical use. This is particularly the case for NLP with the development of large pre-trained models (e.g. GPT, BERT) that are expected to generalize well on a variety of tasks. While the community mainly focused on developing new datasets and metrics, there has been little interest in the aggregation procedure, which is often reduced to a simple average over various performance measures. However, this procedure can be problematic when the metrics are on a different scale, which may lead to spurious conclusions. This paper proposes a new procedure to rank systems based on their performance across different tasks. Motivated by the social choice theory, the final system ordering is obtained through aggregating the rankings induced by each task and is theoretically grounded. We conduct extensive numerical experiments (on over 270k scores) to assess the soundness of our approach both on synthetic and real scores (e.g. GLUE, EXTREM, SE-VAL, TAC, FLICKR). In particular, we show that our method yields different conclusions on stateof-the-art systems than the mean-aggregation procedure while being both more reliable and robust.

show abstract

“…• Emotion classification (Dinkar et al, 2020;Garcia et al, 2019;Colombo et al, 2021a;Witon et al, 2018) where score are continuous.…”

Section: B5 Possible Extensionsmentioning

confidence: 99%

What are the best systems? New perspectives on NLP Benchmarking

Colombo¹,

Noiry²,

Irurozki³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In order to handle the general low agreement between annotators, the consensus between the annotators is then built by computing the root mean square (RMS), as made by Dinkar, Colombo, Labeau, and Clavel (2020).…”

Section: Scoresmentioning

confidence: 99%

Introducing the 3MT_French Dataset to Investigate the Timing of Public Speaking Judgements

Biancardi

Chollet

Clavel

2022

Preprint

Self Cite

View full text Add to dashboard Cite

In most public speaking datasets, judgements are given after watching the entire performance, or on thin slices randomly selected from the presentations, without focusing on the temporal location of these slices. This does not allow to investigate how people's judgements develop over time during presentations. This contrasts with primacy and recency theories, which suggest that some moments of the speech could be more salient than others and contribute disproportionately to the perception of the speaker's performance.To provide novel insights on this phenomenon, we present the 3MT_French dataset. It contains a set of public speaking annotations collected on a crowd-sourcing platform through a novel annotation scheme and protocol. Global evaluation, persuasiveness, perceived self-confidence of the speaker and audience engagement were annotated on different time windows (i.e., the beginning, middle or end of the presentation, or the full video). This new resource will be useful to researchers working on public speaking assessment and training. It will allow to fine-tune the analysis of presentations under a novel perspective relying on socio-cognitive theories rarely studied before in this context, such as first impressions and primacy and recency theories. An exploratory correlation analysis on the annotations provided in the dataset suggests that the early moments of a presentation have a stronger impact on the judgements.

show abstract

“…Although achieving state-of-the-art (SOTA) results on written benchmarks (Wang et al, 2018), they are not tailored to spoken dialog (SD). Indeed, Tran et al (2019) have suggested that training a parser on conversational speech data can improve results, due to the discrepancy between spoken and written language (e.g., disfluencies (Stolcke and Shriberg, 1996), fillers (Shriberg, 1999;Dinkar et al, 2020), different data distribution). Furthermore, capturing discourse-level features, which distinguish dialog from other types of text (Thornbury and Slade, 2006), e.g., capturing multi-utterance dependencies, is key to embed dialog that is not explicitly present in pre-training objectives (Devlin et al, 2018;Yang et al, 2019;Liu et al, 2019), as they often treat sentences as a simple stream of tokens.…”

Section: Introductionmentioning

confidence: 99%

Hierarchical Pre-training for Sequence Labelling in Spoken Dialog

Chapuis

Colombo

Manica

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

Self Cite

View full text Add to dashboard Cite

Sequence labelling tasks like Dialog Act and Emotion/Sentiment identification are a key component of spoken dialog systems. In this work, we propose a new approach to learn generic representations adapted to spoken dialog, which we evaluate on a new benchmark we call Sequence labellIng evaLuatIon benChmark fOr spoken laN-guagE benchmark (SILICONE). SILICONE is model-agnostic and contains 10 di↵erent datasets of various sizes. We obtain our representations with a hierarchical encoder based on transformer architectures, for which we extend two well-known pretraining objectives. Pre-training is performed on OpenSubtitles: a large corpus of spoken dialog containing over 2.3 billion of tokens. We demonstrate how hierarchical encoders achieve competitive results with consistently fewer parameters compared to state-of-the-art models and we show their importance for both pre-training and finetuning.

show abstract

The importance of fillers for text representations of speech transcripts

Cited by 3 publications

References 0 publications

What are the best systems? New perspectives on NLP Benchmarking

What are the best systems? New perspectives on NLP Benchmarking

Introducing the 3MT_French Dataset to Investigate the Timing of Public Speaking Judgements

Hierarchical Pre-training for Sequence Labelling in Spoken Dialog

Contact Info

Product

Resources

About