Extracting and analyzing time-series HCI data from screen-captured task videos

Bao, Lingfeng; Li, Jing; Xing, Zhenchang; Wang, Xinyu; Xia, Xin; Zhang, Bo

doi:10.1007/s10664-015-9417-1

Cited by 29 publications

(8 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In software engineering, there are many works on visual artifacts including recording, tutorials, bug reports [27,29,32,38,61,78,82] with some of them specifically for usability and accessibility testing [30,49,76,81]. In detail, Bao et al [20] focused on the extraction of user interactions to facilitate behavioral analysis of developers during programming tasks, such as code editing, text selection, and window scrolling. Krieter et al [44] analyzed every single frame of a video to generate log files that describe what events are happening at the app level (e.g., the "k" key is pressed).…”

Section: Bug Replay From Visual Informationmentioning

confidence: 99%

GIFdroid: Automated Replay of Visual Bug Reports for Android Apps

Feng¹,

Chen²

2021

Preprint

View full text Add to dashboard Cite

Bug reports are vital for software maintenance that allow users to inform developers of the problems encountered while using software. However, it is difficult for non-technical users to write clear descriptions about the bug occurrence. Therefore, more and more users begin to record the screen for reporting bugs as it is easy to be created and contains detailed procedures triggering the bug. But it is still tedious and time-consuming for developers to reproduce the bug due to the length and unclear actions within the recording. To overcome these issues, we propose GIFdroid, a lightweight approach to automatically replay the execution trace from visual bug reports. GIFdroid adopts image processing techniques to extract the keyframes from the recording, map them to states in GUI Transitions Graph, and generate the execution trace of those states to trigger the bug. Our automated experiments and user study demonstrate its accuracy, efficiency, and usefulness of the approach. CCS CONCEPTS• Software and its engineering → Software testing and debugging.

show abstract

Section: Bug Replay From Visual Informationmentioning

confidence: 99%

GIFdroid: Automated Replay of Visual Bug Reports for Android Apps

Feng¹,

Chen²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Prior approaches exist to derive approximated game logs from gameplay video. However, the majority of these approaches rely on hand-authored event definitions [15] or a secondary machine vision library like OpenCV [2] and access to all of the images in the game [1,10]. This latter approach is only viable for two-dimensional games with limited sets of possible images for each game component (called a "sprite").…”

Section: Related Workmentioning

confidence: 99%

“…We make available a public dataset of player actions for the e-sports game DOTA2 as a secondary contribution in this paper. 1…”

Section: Introductionmentioning

confidence: 99%

Making CNNs for video parsing accessible

Luo

Guzdial

Riedl

2019

Proceedings of the 14th International Conference on the Foundations of Digital Games

View full text Add to dashboard Cite

The ability to extract sequences of game events for high-resolution e-sport games has traditionally required access to the game's engine. This serves as a barrier to groups who don't possess this access. It is possible to apply deep learning to derive these logs from gameplay video, but it requires computational power that serves as an additional barrier. These groups would benefit from access to these logs, such as small e-sport tournament organizers who could better visualize gameplay to inform both audience and commentators. In this paper we present a combined solution to reduce the required computational resources and time to apply a convolutional neural network (CNN) to extract events from e-sport gameplay videos. This solution consists of techniques to train a CNN faster and methods to execute predictions more quickly. This expands the types of machines capable of training and running these models, which in turn extends access to extracting game logs with this approach. We evaluate the approaches in the domain of DOTA2, one of the most popular e-sports. Our results demonstrate our approach outperforms standard backpropagation baselines.

show abstract

“…We find that the perceptions of developers about comparable technologies and the choices they make which technology to use are very likely to be influenced by how other developers see and evaluate the technologies. So developers often turn to the two information sources on the Web [4] to learn more about comparable technologies.…”

Section: Introductionmentioning

confidence: 99%

DiffTech: a tool for differencing similar technologies from question-and-answer discussions

Wang

Chen

Xing

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

Self Cite

View full text Add to dashboard Cite

Developers can use different technologies for different software development tasks in their work. However, when faced with several technologies with comparable functionalities, it can be challenging for developers to select the most appropriate one, as trial and error comparisons among such technologies are time-consuming. Instead, developers resort to expert articles, read official documents or ask questions in Q&A sites for technology comparison. However, it is still very opportunistic whether they will get a comprehensive comparison, as online information is often fragmented, contradictory and biased. To overcome these limitations, we propose the DiffTech system that exploits the crowd sourced discussions from Stack Overflow, and assists technology comparison with an informative summary of different comparison aspects. We found 19,118 comparative sentences from 2,410 pairs of comparable technologies. We released our DiffTech website for public use. Our website attracts over 1800 users and we also receive some positive comments on social media. A walkthrough video of the tool demo: https:// www.youtube.com/ watch?v=ixX41DXRNsI Website link: https:// difftech.herokuapp.com/ CCS CONCEPTS • Software and its engineering;

show abstract

Extracting and analyzing time-series HCI data from screen-captured task videos

Cited by 29 publications

References 65 publications

GIFdroid: Automated Replay of Visual Bug Reports for Android Apps

GIFdroid: Automated Replay of Visual Bug Reports for Android Apps

Making CNNs for video parsing accessible

DiffTech: a tool for differencing similar technologies from question-and-answer discussions

Contact Info

Product

Resources

About