RoScript

Qian, Ju; Shang, Zhengyu; Yan, Shuoyan; Wang, Yan; Lin, Chen

doi:10.1145/3377811.3380431

Cited by 28 publications

(5 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yeh et al introduced a system utilizing GUI screenshots for visual search and automated graphical user interfaces [7]. Qian et al harnessed computer vision techniques for robotically operating applications through a touch screen, automating GUI testing [8]. Chen et al introduced a deep learning-based method encoding visual and text information to recover missing labels for the current UI, simplifying their retrieval via text queries [9].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention Mechanism

Pan,

Huan,

et al. 2024

Applied Sciences

View full text Add to dashboard Cite

Graphical user interfaces (GUIs) hold an irreplaceable position in modern software and applications. Users can interact through them. Due to different terminal devices, there are sometimes display errors, such as component occlusion, image loss, text overlap, and empty values during software rendering. To address the aforementioned common four GUI display errors, a target detection algorithm based on the improved Faster R-CNN is proposed. Specifically, ResNet-50 is used instead of the traditional VGG-16 as the feature extraction network. The feature pyramid network (FPN) and the enhanced multi-scale attention (EMA) algorithm are introduced to improve accuracy. ROI-Align is used instead of ROI-Pooling to enhance the generalization capability of the network. Since training models require a large number of labeled screenshots of errors, there is currently no publicly available dataset with GUI display problems. Therefore, a training data generation algorithm has been developed, which can automatically generate screenshots with GUI display problems based on the Rico dataset. Experimental results show that the improved Faster R-CNN achieves a detection accuracy of 87.3% in the generated GUI problem dataset, which is a 7% improvement compared to the previous version.

show abstract

Section: Related Workmentioning

confidence: 99%

“…To explain the meaning of the aforementioned metric, the concepts of precision and recall are introduced. The formulas for precision and recall are shown in Equation (8).…”

Section: Evaluation Metricsmentioning

confidence: 99%

Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention Mechanism

Pan,

Huan,

et al. 2024

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…In 2016, Yu et al proposed an image-based GUI automation testing method that utilizes image recognition technology to identify elements in GUI images and simulates input signals from the System Under Test (SUT) using input devices [48]. Subsequently, Ju et al designed a non-intrusive script-driven robot testing system specifically for automating the testing of touchscreen applications [16]. This method employs visual test scripts to express GUI operations on the touchscreen application and utilizes a physical robot to drive the automated test execution.…”

Section: Non-intrusive Gui Testing Systemmentioning

confidence: 99%

“…This strategy is advantageous in distinguishing features within compact text regions, particularly narrow character features. The text recognition branch is structured with a VGG-type convolutional network, a bidirectional LSTM [16,42], a fully connected network, and lastly, a Connectionist Temporal Classification (CTC) decoder. Table 1 provides a detailed representation of the text recognition structure.…”

Section: Text Recognitionmentioning

confidence: 99%

“…It does not rely on hardware interfaces and employs cameras to capture images of the vehicle's center control screen for evaluation. RoScript [16] employs visual testing scripts to articulate GUI operations on touchscreen applications, subsequently utilizing physical robots to drive the automation of test execution. NiCro [17] incorporates the most advanced GUI widget detectors to identify widgets from GUI images, subsequently analyzing a composite set of information to match widgets across different devices.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Non-Intrusive Automated Testing System for Internet of Vehicles App Based on Deep Learning

Zhang

Sun

2023

Electronics

View full text Add to dashboard Cite

In the non-intrusive automated testing system for Internet of Vehicles (IoV) applications, automatic recognition of text and icons on vehicle central control screens is of paramount importance. However, the detection and recognition of content on vehicle central control screens are inherently complex. Additionally, during non-intrusive vehicle central control screen image testing, there is a deficiency of suitable datasets and detection methods. This deficiency renders information within vehicle application images difficult to be accurately extracted by the detection network. To address this problem, this study first constructs a dataset tailored for text detection and recognition on vehicle screens. This dataset encompasses a variety of vehicle central control images, enabling the generic text detection and recognition network to more effectively identify and interpret text within vehicle screens. Subsequently, this research proposes an enhanced Fully Convolutional Networks for Text Detection (FOTS) method for vehicle central control screen text detection and recognition. This method elevates the semantic expression capabilities of features by sharing vehicle central control screen text detection and recognition features. Furthermore, it improves multi-scale feature processing capabilities through the utilization of a feature transformation module. Validation through visual and quantitative experiments demonstrates that the proposed method can effectively accomplish text detection and recognition tasks on vehicle screens. This achievement bears significant implications for the field of automated testing in IoV applications.

show abstract

Sketch2Wireframe: an automatic framework for transforming hand-drawn sketches to digital wireframes in UI design

Liang,

Lin

2023

Vis Comput

View full text Add to dashboard Cite

RoScript

Cited by 28 publications

References 32 publications

Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention Mechanism

Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention Mechanism

A Non-Intrusive Automated Testing System for Internet of Vehicles App Based on Deep Learning

Sketch2Wireframe: an automatic framework for transforming hand-drawn sketches to digital wireframes in UI design

Contact Info

Product

Resources

About