End-to-end let's play commentary generation using multi-modal video representations

Li, Chengxi; Gandhi, Sagar; Harrison, Brent

doi:10.1145/3337722.3341870

Cited by 8 publications

(3 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some works targeted the automated generation of a comprehensive description of what happens in gameplay videos (i.e., game commentary). Examples of these works are the framework by Guzdial et al [11] and the approach presented by Li et al [15] modeling the generation of commentaries as a sequence-to-sequence problem, converting video clips to commentary. On the same line of research, Shah et al [24] presented an approach to generate automatic comments for videos by using deep convolutional neural networks.…”

Section: Mining Of Gameplay Videosmentioning

confidence: 99%

Towards Using Gameplay Videos for Detecting Issues in Video Games

Guglielmi¹,

Scalabrino²,

Bavota³

et al. 2022

Preprint

View full text Add to dashboard Cite

Context. The game industry is increasingly growing in recent years. Every day, millions of people play video games, not only as a hobby, but also for professional competitions (e.g., e-sports or speedrunning) or for making business by entertaining others (e.g., streamers). The latter daily produce a large amount of gameplay videos in which they also comment live what they experience. Since no software and, thus, no video game is perfect, streamers may encounter several problems (such as bugs, glitches, or performance issues). However, it is unlikely that they explicitly report such issues to developers. The identified problems may negatively impact the user's gaming experience and, in turn, can harm the reputation of the game and of the producer. Objective. We aim at proposing and empirically evaluating GELID, an approach for automatically extracting relevant information from gameplay videos by (i) identifying video segments in which streamers experienced anomalies; (ii) categorizing them based on their type and context in which appear (e.g., bugs or glitches appearing in a specific level or scene of the game); and (iii) clustering segments that regard the same specific issue. Method. We will build on top of existing approaches able to identify videos that are relevant for a specific video game. These represent the input of GELID that processes them to achieve the defined objectives. We will experiment GELID on several gameplay videos to understand the extent to which each of its steps is effective. CCS CONCEPTS• Software and its engineering → Software evolution; Maintaining software; Software defect analysis.

show abstract

Section: Mining Of Gameplay Videosmentioning

confidence: 99%

Towards Using Gameplay Videos for Detecting Issues in Video Games

Guglielmi¹,

Scalabrino²,

Bavota³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Some works targeted the automated generation of a comprehensive description of what happens in gameplay videos (i.e., game commentary). Examples of these works are the framework by Guzdial et al (2018) and the approach presented by Li et al (2019) modeling the generation of commentaries as a sequence-to-sequence problem, converting video clips to commentary. On the same line of research, an approach to generate automatic comments for videos by using deep convolutional neural networks was presented by Shah et al (2019).…”

Section: Mining Of Gameplay Videosmentioning

confidence: 99%

Sorry, I don’t Understand: Improving Voice User Interface Testing

Guglielmi

Rosa

Scalabrino

et al. 2022

Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering

View full text Add to dashboard Cite

Context. The game industry is increasingly growing in recent years. Every day, millions of people play video games, not only as a hobby, but also for professional competitions (e.g., e-sports or speed-running) or for making business by entertaining others (e.g., streamers). The latter daily produce a large amount of gameplay videos in which they also comment live what they experience. But no software and, thus, no video game is perfect: Streamers may encounter several problems (such as bugs, glitches, or performance issues) while they play. Also, it is unlikely that they explicitly report such issues to developers. The identified problems may negatively impact the user's gaming experience and, in turn, can harm the reputation of the game and of the producer. Objective. In this paper, we propose and empirically evaluate GELID, an approach for automatically extracting relevant information from gameplay videos by (i) identifying video segments in which streamers experienced anomalies; (ii) categorizing them based on their type (e.g., logic or presentation); clustering them based on (iii) the context in which appear (e.g., level or game area) and (iv) on the specific issue type (e.g., game crashes). Method. We manually defined a training set for step 2 of GELID (categorization) and a test set for validating in isolation the four components of GELID. In total, we manually segmented, labeled, and clustered 170 videos related to 3 video games, defining a dataset containing 604 segments. Results. While in steps 1 (segmentation) and 4 (specific issue clustering) GELID achieves satisfactory results, it shows limitations on step 3 (game context clustering) and, above all, step 2 (categorization).

show abstract

“…To the best of our knowledge there have only been two attempts at this problem, the first focused on generation of a bag-of-words representation, which is an unordered collection of words that does not constitute legible commentary (Guzdial, Shah, and Riedl 2018). The second attempt at this problem instead structured commentary generation as a sequence-tosequence generation task (Li, Gandhi, and Harrison 2019). We do not compare against this second approach as it was not yet published during the development of this research.…”

Section: Introductionmentioning

confidence: 99%

Automated Let's Play Commentary

Shah,

Guzdial,

Riedl

2019

Preprint

View full text Add to dashboard Cite

Let's Plays of video games represent a relatively unexplored area for experimental AI in games. In this short paper, we discuss an approach to generate automated commentary for Let's Play videos, drawing on convolutional deep neural networks. We focus on Let's Plays of the popular game Minecraft. We compare our approach and a prior approach and demonstrate the generation of automated, artificial commentary.

show abstract

End-to-end let's play commentary generation using multi-modal video representations

Cited by 8 publications

References 14 publications

Towards Using Gameplay Videos for Detecting Issues in Video Games

Towards Using Gameplay Videos for Detecting Issues in Video Games

Sorry, I don’t Understand: Improving Voice User Interface Testing

Automated Let's Play Commentary

Contact Info

Product

Resources

About