SemEval-2017 Task 6: #HashtagWars: Learning a Sense of Humor

Abstract

This paper describes a new shared task for humor understanding that attempts to eschew the ubiquitous binary approach to humor detection and focus on comparative humor ranking instead. The task is based on a new dataset of funny tweets posted in response to shared hashtags, collected from the 'Hashtag Wars' segment of the TV show @midnight. The results are evaluated in two subtasks that require the participants to generate either the correct pairwise comparisons of tweets (subtask A), or the correct ranking of the tweets (subtask B) in terms of how funny they are. 7 teams participated in subtask A, and 5 teams participated in subtask B. The best accuracy in subtask A was 0.675. The best (lowest) rank edit distance for subtask B was 0.872.

DataStories at SemEval-2017 Task 6: Siamese LSTM with Attention for Humorous Text Comparison

“…Furthermore, we compare the performance of our system on the #HastagWars dataset (Potash et al, 2016). Table 3 shows that our improved model outperforms the other approaches.…”

“…Table 3 shows that our improved model outperforms the other approaches. The reported results are the average of 3 Leave-One-Out runs, in order to be comparable with (Potash et al, 2016). Figure 3 shows the detailed results of our model on the #HastagWars dataset, with the accuracy distribution over the hashtags.…”

“…System Acc Micro Avg LSTM (token) (Potash et al, 2016) 0.554 (± 0.0085) CNN (char) (Potash et al, 2016) 0.637 (± 0.0074) DataStories (fixed) 0.696 (± 0.0075) Experimental Setup. For developing our models we used Keras (Chollet, 2015), Theano (Theano Dev Team, 2016) and Scikit-learn (Pedregosa et al, 2011).…”

“…Most computational humor tasks focus on the problem of humor detection. However SemEval-2017 Task 6 (Potash et al, 2017) explores the subjective nature of humor, using a dataset of Twitter messages posted in the context of the TV show "@midnight". At each episode during the segment "Hashtag Wars", a topic in the form of a hashtag is given and viewers of the show post funny tweets including that hashtag.…”

SRHR at SemEval-2017 Task 6: Word Associations for Humour Recognition

“…We performed all training and testing on the dataset introduced in Potash et al (2017) specifically for this task. The dataset consists of response tweets to 112 hashtags created by @midnight.…”