Deep unsupervised multi-view detection of video game stream highlights

Abstract

We consider the problem of automatic highlight-detection in video game streams. Currently, the vast majority of highlight-detection systems for games are triggered by the occurrence of hard-coded game events (e.g., score change, end-game), while most advanced tools and techniques are based on detection of highlights via visual analysis of game footage. We argue that in the context of game streaming, events that may constitute highlights are not only dependent on game footage, but also on social signals that are conveyed by the streamer during the play session (e.g., when interacting with viewers, or when commenting and reacting to the game). In this light, we present a multi-view unsupervised deep learning methodology for novelty-based highlight detection. The method jointly analyses both game footage and social signals such as the players facial expressions and speech, and shows promising results for generating highlights on streams of popular games such as Player Unknown's Battlegrounds.

From Pixels to Affect: A Study on Games and Player Experience

“…At the same time, the most common approaches for analysing player experience, besides game and gameplay information, heavily rely on direct measurements from players, such as face monitoring, speech and physiological signals; see e.g. [4], [36], [39]). Unlike these approaches, our methodology relies solely of gameplay video information.…”

Intelligent Video Highlights Generation with Front-Camera Emotion Sensing

“…As mentioned in Section 2, there are a number of recent works which have attempted to utilize emotion in addition to video features to generate highlights. Works such as [43,44] produce high quality highlights on football videos and video game streams, but may not perform well on more varied videos, as selected in this work. In addition, [43] requires human input to record emotion, and [44] is designed specifically for streaming setups.…”

“…Fiao et al [43] also uses emotions along with video to generate highlights; however, they focus on sports videos, and requires external hardware and human interaction to gather emotion data. Ringer et al [44] used deep learning to measure emotion of a face camera during streaming of video games; this work is restricted to highlight the generation of video games, which similarly to sports, is strongly correlated with audio energy. Kaklauskas et al [45,46] study specifically video ads, and require a plethora of hardware and human interaction to gather various physiological signals.…”

Modeling Individual and Team Behavior through Spatio-temporal Analysis

“…Thus, the labels were revised and a final set of labels was produced consisting of seven labels: "Pickaxing for Gold"; "Exploring"; "Tunneling -Pickaxe", the player uses a pickaxe to get to another area or clear an area for a mine; "Tunneling -Explosive", a player uses an explosive to get to another area or clear an area for a mine; "Explosive -Inner", a (a) (b) Figure 6. Sequence graph showing (a) high-performing team players in colored nodes and (b) players from one high-performing (25,26,27,28,29) and one low-performing (30,31,32) team.…”