Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements

Abstract

Natural language descriptions of user interface (UI) elements such as alternative text are crucial for accessibility and language-based interaction in general. Yet, these descriptions are constantly missing in mobile UIs. We propose widget captioning, a novel task for automatically generating language descriptions for UI elements from multimodal input including both the image and the structural representations of user interfaces. We collected a largescale dataset for widget captioning with crowdsourcing. Our dataset contains 162,859 language phrases created by human workers for annotating 61,285 UI elements across 21,750 unique UI screens. We thoroughly analyze the dataset, and train and evaluate a set of deep model configurations to investigate how each feature modality as well as the choice of learning strategies impact the quality of predicted captions. The task formulation and the dataset as well as our benchmark models contribute a solid basis for this novel multimodal captioning task that connects language and user interfaces. * Equal contribution † Participated in the project during an internship at Google Research.

AppBuddy: Learning to Accomplish Tasks in Mobile Apps via Reinforcement Learning

“…However, we had to manually add meaningful textual descriptions to the view hierarchy to do so. Fortunately, adding such descriptions can be done automatically by using a widget captioning technique [25]. This is a promising direction for future work.…”

UIBert: Learning Generic Multimodal Representations for UI Understanding

“…For example, [Li et al, 2020b] leveraged Transformer to map natural language commands to executable actions in a UI. [Li et al, 2020c; used Transformer to generate textual descriptions for UI elements. There were also attempts using convolutional neural networks to retrieve similar UIs for design mining [Deka et al, 2017;Liu et al, 2018;Huang et al, 2019].…”

“…[Li et al, 2020c; used Transformer to generate textual descriptions for UI elements. There were also attempts using convolutional neural networks to retrieve similar UIs for design mining [Deka et al, 2017;Liu et al, 2018;Huang et al, 2019]. Past work generally built task-specific models and required substantial labeled data.…”

“…Previous work usually utilized single-modality data, e.g. only image, to solve the tasks [Liu et al, 2018;. How to effectively leverage cross-modal knowledge and diminish the affect of noise for general UI understanding remains an open question.…”

“…How to effectively leverage cross-modal knowledge and diminish the affect of noise for general UI understanding remains an open question. Second, high-quality task-specific UI data is expensive to achieve as it requires complicated setups of app/web crawlers and time-consuming human labeling work [Li et al, 2020c;Swearngin and Li, 2019], which inevitably slows the model development cycle. When large-scale data is unavailable, it's non-trivial to overcome overfitting and achieve satisfying performance.…”