A Novel approach of GUI Mapping with image based widget detection and classification

Widget recognition is crucial for automated Android black box testing. Over the past ten years, different industrial tools and academic works have been available for identifying Graphical User Interface (GUI) components in Android screens. Traditional identification methods, like GUI hierarchy parsing, often struggle with dynamic content and complex structures. In contrast, Computer Vision (CV) techniques provide greater robustness and flexibility to adapt to different screen resolutions, design specifications, and patterns. However, the CV-based solutions available are still limited concerning the variety of widgets that can be recognized. Moreover, the current identification of GUI components mainly relies on classification, which can lead to ambiguous lists with repeated elements. In this paper, we combine different CV-based techniques to extract context-based descriptions for each widget, to enhance the identification process by going beyond class recognition for describing widgets. We also implemented two primary CV-based approaches for widget recognition: Object Detection combined with Classification, and a One-Stage Recognition method. We trained and evaluated the approaches on a custom 105 classes widget dataset. Moreover, we present a Computer Vision-based method for describing widgets using their contextual meaning on Android screen captures.

show abstract

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

customersupport@researchsolutions.com

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

A Novel approach of GUI Mapping with image based widget detection and classification

Cited by 2 publications

References 10 publications

Construction of Semantic Model for GUI of Mobile Applications Using Deep Learning

Construction of Semantic Model for GUI of Mobile Applications Using Deep Learning

Enhancing widget recognition for automated Android testing using Computer Vision techniques

Contact Info

Product

Resources

About