Translating video recordings of mobile app usages into replayable scenarios

Abstract

Screen recordings of mobile applications are easy to obtain and capture a wealth of information pertinent to software developers (e.g., bugs or feature requests), making them a popular mechanism for crowdsourced app feedback. Thus, these videos are becoming a common artifact that developers must manage. In light of unique mobile development constraints, including swift release cycles and rapidly evolving platforms, automated techniques for analyzing all types of rich software artifacts provide benet to mobile developers. Unfortunately, automatically analyzing screen recordings presents serious challenges, due to their graphical nature, compared to other types of (textual) artifacts. To address these challenges, this paper introduces V2S, a lightweight, automated approach for translating video recordings of Android app usages into replayable scenarios. V2S is based primarily on computer vision techniques and adapts recent solutions for object detection and image classication to detect and classify user actions captured in a video, and convert these into a replayable test scenario. We performed an extensive evaluation of V2S involving 175 videos depicting 3,534 GUI-based actions collected from users exercising features and reproducing bugs from over 80 popular Android apps. Our results illustrate that V2S can accurately replay scenarios from screen recordings, and is capable of reproducing ⇡ 89% of our collected videos with minimal overhead. A case study with three industrial partners illustrates the potential usefulness of V2S from the viewpoint of developers. CCS CONCEPTS • Software and its engineering → Software maintenance tools; Software verication and validation; Application specic development environments.

It Takes Two to Tango: Combining Visual and Textual Information for Detecting Duplicate Video-Based Bug Reports

“…We collected video-based bug reports for six open-source Android apps, namely AntennaPod (APOD) [2], Time Tracker (TIME) [6], Token (TOK) [7], GNUCash (GNU) [4], Grow-Tracker (GROW) [5], and Droid Weight (DROID) [3]. We selected these apps because they have been used in previous studies [25,34,78,79], support different app categories (finance, productivity, etc. ), and provide features that involve a variety of GUI interactions (taps, long taps, swipes, etc.).…”

“…The analysis of graphical software artifacts to support software engineering tasks has been common in recent years. Such tasks include mobile app testing [25,56,61,79], developer/user behavior modelling [22,30,48], GUI reverse engineering and code generation [24,38,39,44,80,83], analysis of programming videos [20,69,76,85,103,104], and GUI understanding and verification [33,105]. None of these works deal with finding duplicate video-based bug reports, which is our focus.…”

“…We conducted two empirical studies to measure: (i) the effectiveness of different configurations of TANGO by examining the benefit of combining visual and textual information from videos, as opposed to using only a single information source; and (ii) Ta n g o 's ability to save developer effort in identifying duplicate video-based bug reports. To carry out these studies, we collected a set of 180 video-bug reports from six popular apps used in prior research [25,34,78,79], and defined 4,860 duplicate detection tasks that resemble those that developers currently face for textual bug reports -wherein a corpus of potential duplicates must be ranked according to their similarity to an incoming bug report.…”

Layout and Image Recognition Driving Cross-Platform Automated Mobile Testing

“…Qian et al [35] propose a tool leveraging visual test scripts to express GUI actions and using a physical robot to drive automated test execution. Bernal-Cardenas et al [36] introduce V2S to translate video recordings of Android app usages into replayable scenarios.…”

GIFdroid: Automated Replay of Visual Bug Reports for Android Apps

“…However, those approaches do not apply to video-based bug reports. There are platforms providing both video recording and replaying functionalities [1,23,45,62] which also store the low-level program execution information. They require the framework installation or app instrumentation which is too heavy for end users.…”