Grounding truth via ordinal annotation

Yannakakis, Georgios N.; Martínez, Héctor

doi:10.1109/acii.2015.7344627

Cited by 56 publications

(63 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Indeed, this has been shown elsewhere in the literature (Yang and Chen, 2011;Yannakakis and Martínez, 2015a;Parthasarathy et al, 2016), where rank-based annotations of affect dimensions yield higher inter-rater reliability than conventional absolute ratings. In Yannakakis and Martínez (2015a), evaluators only rank (i.e., increase or decrease their rating) when they perceive changes in affect dimensions. This facilitates the annotation process in terms of inter-rater agreement and reduction of cognitive load, because the ranking scheme allows simplification of annotation as well as being robust to different personal scales in affect dimension of evaluators.…”

Section: Related Workmentioning

confidence: 75%

“…Firstly, an analysis in a continuously annotated database showed that evaluators tend to agree more on relative emotions (e.g., an increase in arousal) than on absolute emotions (e.g., arousal) (Metallinou and Narayanan, 2013). Indeed, this has been shown elsewhere in the literature (Yang and Chen, 2011;Yannakakis and Martínez, 2015a;Parthasarathy et al, 2016), where rank-based annotations of affect dimensions yield higher inter-rater reliability than conventional absolute ratings. In Yannakakis and Martínez (2015a), evaluators only rank (i.e., increase or decrease their rating) when they perceive changes in affect dimensions.…”

Section: Related Workmentioning

confidence: 96%

“…Despite a trend toward evaluators being encouraged to rate when changes in affect dimensions occur (Yannakakis and Martínez, 2015a;Celiktutan and Gunes, 2016), to the best of our knowledge, there are no studies introducing annotation of emotion changes in dimension ratings apart from Yannakakis and Martínez (2015a). Ordinal or ranked-based annotations such as increase and decrease of arousal and valence ratings have shown improved inter-rater reliability over conventional absolute ratings (Yannakakis and Martínez, 2015a).…”

Section: Delta Emotion Ground Truthmentioning

confidence: 99%

See 2 more Smart Citations

Prediction of Emotion Change From Speech

Huang

Epps

2018

Front. ICT

View full text Add to dashboard Cite

The fact that emotions are dynamic in nature and evolve across time has been explored relatively less often in automatic emotion recognition systems to date. Although withinutterance information about emotion changes recently has received some attention, there remain open questions unresolved, such as how to approach delta emotion ground truth, how to predict the extent of emotion change from speech, and how well change can be predicted relative to absolute emotion ratings. In this article, we investigate speech-based automatic systems for continuous prediction of the extent of emotion changes in arousal/valence. We propose the use of regression (smoothed) deltas as ground truth for emotion change, which yielded considerably higher inter-rater reliability than first-order deltas, a commonly used approach in previous research, and represent a more appropriate approach to derive annotations for emotion change research, findings which are applicable beyond speech-based systems. In addition, the first system design for continuous emotion change prediction from speech is explored. Experimental results under the Output-Associative Relevance Vector Machine framework interestingly show that changes in emotion ratings may be better predicted than absolute emotion ratings on the RECOLA database, achieving 0.74 vs. 0.71 for arousal and 0.41 vs. 0.37 for valence in concordance correlation coefficients. However, further work is needed to achieve effective emotion change prediction performances on the SEMAINE database, due to the large number of non-change frames in the absolute emotion ratings.

show abstract

Section: Related Workmentioning

confidence: 75%

Section: Related Workmentioning

confidence: 96%

Section: Delta Emotion Ground Truthmentioning

confidence: 99%

See 1 more Smart Citation

Prediction of Emotion Change From Speech

Huang

Epps

2018

Front. ICT

View full text Add to dashboard Cite

show abstract

“…Our recommendation is based on studies by Yannakakis and Martínez showing limitations when using the more common choice of ratings for subjective assessment, which provide empirical evidence for the superiority of rank-based questionnaire schemes for subjective annotation (in games and beyond) [20,21,10]. In particular, rating-based questionnaires generate lower inter-rater agreement and higher levels of inconsistency and order effects, and are dominated by a number of critical biases that make any posthoc analysis questionable [21,20,18,11].…”

Section: Which Questionnaire Should I Use?mentioning

confidence: 99%

Evaluating content generators

Shaker

Smith

Yannakakis

2016

Procedural Content Generation in Games

Self Cite

View full text Add to dashboard Cite

Evaluating your content generator is a very important task, but difficult to do well. Creating a game content generator in general is much easier than creating a good game content generator-but what is a "good" content generator? That depends very much on what you are trying to create and why. This chapter discusses the importance and the challenges of evaluating content generators, and more generally understanding a generator's strengths and weaknesses and suitability for your goals. In particular, we discuss two different approaches to evaluating content generators: visualizing the expressive range of generators, and using questionnaires to understand the impact of your generator on the player. These methods could broadly be called top-down and bottom-up methods for evaluating generators. I created a generator, now what?The entirety of this book thus far has been focused on how to create procedural content generators, using a variety of techniques and for many different purposes. We hope that, by now, you have gained an appreciation for the strengths and weaknesses of different approaches to PCG, and also the surprises that can come from writing a generative system. We imagine that you also have experienced some of the frustration that can come from debugging a generative system: "is the interesting level I created a fluke, a result of a bug, or a genuine result?"Creating a generator is one thing; evaluating it is another. Regardless of the method followed, generators are evaluated on their ability to achieve the desired goals of the designer (or the computational designer). This chapter reviews methods for achieving that. Arguably, the generation of any content is trivial; the generation of valuable content for the task at hand, on the other hand, is a rather challenging procedure. Further, it is more challenging to generate content that is both valuable and novel.

show abstract

“…Which facial expression looks happier?). Recent comparative studies have argued that rating approaches have disadvantages compared to ranking questionnaire schemes [32,16], such as increased order-ofplay and inconsistency effects [30] and lower inter-rater agreement [17,31].…”

Section: Model Output: Experience Annotationmentioning

confidence: 99%

The experience-driven perspective

Shaker

Togelius

Yannakakis

2016

Procedural Content Generation in Games

Self Cite

View full text Add to dashboard Cite

Ultimately, content is generated for the player. But so far, our algorithms have not taken specific players into account. Creating computational models of a player's behaviour, preferences, or skills is called player modelling. With a model of the player, we can create algorithms that create content specifically tailored to that player. The experience-driven perspective on procedural content generation provides a framework for content generation based on player modelling; one of the most important ways of doing this is to use a player model in the evaluation function for search-based PCG. This chapter discusses different ways of collecting and encoding data about the player, primarily player experience, and ways of modelling this data. It also gives examples of different ways in which such models can be used.

show abstract

Grounding truth via ordinal annotation

Cited by 56 publications

References 19 publications

Prediction of Emotion Change From Speech

Prediction of Emotion Change From Speech

Evaluating content generators

The experience-driven perspective

Contact Info

Product

Resources

About