YOLOv5-MGC: GUI Element Identification for Mobile Applications Based on Improved YOLOv5

Abstract

The identification of interface elements is the first step in mobile application automated testing and the key to smooth testing. However, existing object detection algorithms have a low accuracy rate, and some tiny elements are missed in the recognition of graphical user interface (GUI) elements. To address this limitation, this paper proposes the YOLOv5-MGC algorithm, a robot vision-based interface element recognition algorithm for mobile applications. The algorithm improves the network by using K-means++ algorithm for target anchor box generation, applying the attention mechanism, adding a microscale detection layer, and introducing the Ghost bottleneck module. The proposed approach enhances the recognition accuracy of the elements through the target anchor box and attention mechanism. Moreover, it enhances the network’s ability to detect tiny elements, which improves the shortcomings of the current target detection algorithm and is conducive to further promoting mobile application robot testing and enhancing robot testing automation. Experimental results show that the YOLOv5-MGC algorithm is superior to the YOLOv5 for object detection in the recognition of GUI elements, with the mean average precision (mAP_0.5) reaching 89.8% and the recognition precision reaching 80.8%.

Finite State GUI Testing with Test Case Prioritization Using Z-BES and GK-GRU

“…The YOLOv5 algorithm that enhanced the mobile app interface element recognition significantly improved the analysis accuracy and identification of minute components [17]. The experimental findings confirmed its superiority in GUI element identification, demonstrating promise for future development in robot testing automation for mobile applications.…”

Improved Detector Based on Yolov5 for Typical Targets on the Sea Surfaces

“…The basic model selected for our detection task contains several changes based on the yolov5m6 [12] baseline model. Here '6′ denotes 'P6′, which means an extra P6 object output layer is added in the yolov5-P6 models for the detection of larger objects.…”

“…Other two-stage detectors, such as R-FCN [8], Cascade R-CNN [9], and Mask R-CNN [10] extended on the basis of R-CNN or its variant, can also gain remarkable performance in object detection tasks. Single-stage algorithms such as the You-Only-Look-Once (YOLO) series [11,12], single-shot multi-box detector (SSD) [13], and RetinaNet [14] can usually run at much higher speeds at inference. YOLOv1 [11] provided unified training for classification and localization, reaching an inference speed of 45 fps (much faster than Faster R-CNN) and a mean average precision (mAP) score of 63.4% (comparable to Faster R-CNN (ZF version)) evaluated on the PASCAL VOC [15] 2007 and 2012 datasets.…”

“…Known for its fast processing speed and high accuracy, the YOLOv5 model [12] has been employed extensively in a wide range of areas and real-world applications in recent years. For instance, Wang et al [16] constructed a remote sensing super-resolution object detection dataset that provided public access to data for various classes with small targets and proposed the multi-class cyclic generative adversarial network with residual feature aggregation with YOLOv5 detector, which significantly enhanced the detection performance of small objects.…”