A Survey on the Use of Computer Vision to Improve Software Engineering Tasks

Bajammal, Mohammad; Stocco, Andrea; Mazinanian, Davood; Mesbah, Ali

doi:10.1109/tse.2020.3032986

Cited by 11 publications

(6 citation statements)

References 104 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As previously outlined, existing automated web testing techniques and tools do not work for the <canvas>, but prior research has also indicated that <canvas> bugs overlap with visual bugs found in graphical user interfaces (GUIs) and generic web applications [23]. We refer to the survey of computer vision applications in software engineering by Bajammal et al [5] and the grey literature review of AI-based test automation techniques by Ricca et al [36] for an overview of visual testing for GUIs and web apps. Here, we only discuss related work that was not covered in the survey by Bajammal et al [5].…”

Section: Visual Web and Gui Testingmentioning

confidence: 99%

“…We refer to the survey of computer vision applications in software engineering by Bajammal et al [5] and the grey literature review of AI-based test automation techniques by Ricca et al [36] for an overview of visual testing for GUIs and web apps. Here, we only discuss related work that was not covered in the survey by Bajammal et al [5].…”

Section: Visual Web and Gui Testingmentioning

confidence: 99%

See 1 more Smart Citation

Automatically Detecting Visual Bugs in HTML5 Games

Macklon¹,

Taesiri²,

Viggiato³

et al. 2022

Preprint

View full text Add to dashboard Cite

The HTML5

show abstract

Section: Visual Web and Gui Testingmentioning

confidence: 99%

Section: Visual Web and Gui Testingmentioning

confidence: 99%

Automatically Detecting Visual Bugs in HTML5 Games

Macklon¹,

Taesiri²,

Viggiato³

et al. 2022

Preprint

View full text Add to dashboard Cite

The HTML5

show abstract

“…Pure image pixel comparison is ineffective. We expect that the robot can focus (i) Input: recorded video RV (ii) Output: test sequence script TSS (1) split_tag � 0 (2) for each frame in RV do (3) finger � getOpenPoseDetection(RV) (4) if finger is exist then (5) gesture_info ⇐ map(frame, split_tag, finger) ( 6) else (7) gui_info ⇐ map(frame, split_tag) (8) if pre_frames is gesture frame then (9) split_tag ++ (10) end if (11) end if (12) end for (13) for st � 0 to split_tag do (14) gui � getMiddleFrame(gui_info, st) (15) gui_elements � getObjectDetection(gui) (16) gui_skeleton � getGuiSkeleton(gui_elements) (17) gesture_micro � identifyGesture(gesture_info, st) (18) object � identifyObject(gesture_micro, gui_elements) (19) TTS ⇐ getTestSquence(st, gui_skeleton, gesture_micro, object) (20) end for (21) gui_elements' � getObjectDetection(gui_current) (5) gui_skeleton' � getGUISkeleton(gui_elements') (6) if serial_number is 0 then (7) break (8) else if serial_number is 1 then (9) for each tss in TSSs do (10) gui_skeleton � getTSSGUI(tss, serial_number) (11) e � getSimilarityJudgment(gui_skeleton', gui_skeleton) (12) end for (13) TSS' � getMaxSimliarScript(e, s_threshold) (14) serial_number � execteAction(TSS', serial_number, gui_elements') ( 15) else (16) gui_skeleton � getTSSGUI(TSS′, serial_number) (17) e � getSimilarityJudgment(gui_skeleton', gui_skeleton) (18) if e > s_threshold then (19) TR ⇐ recordResults() (20) serial_number � exectueAction(TSS', serial_number, gui_elements') ( 21) else (2...…”

Section: Gui Identification and Judgementmentioning

confidence: 99%

“…e GUI driven record-relay testing further captures GUI images to replay. With the adoption of computer vision in software engineering [13], GUI images are included in mobile app testing. Sikuli [14] and Eyeautomate [15] can generate scripts that contain screenshots of GUI elements and replay across devices by comparing image pixels of the elements.…”

Section: Introductionmentioning

confidence: 99%

Learning-Replay Based Automated Robotic Testing for Mobile App

Feng

Zhang

2022

Mobile Information Systems

View full text Add to dashboard Cite

Record-replay testing is widely used in mobile app testing as an automated testing method. However, the current record-replay methods are closely dependent on the internal information of the device or app under test. Due to the diversity of mobile devices and system platforms, their practical use is limited. To break this limitation, this paper proposes an entirely black-box learning-replay testing approach by combining robotics and vision technology to achieve a record-replay testing that can support cross-device and cross-platform. Firstly, vision technology is used to extract the critical information of GUI and gesture actions during the tester’s testing process; secondly, the GUI composition and test actions are analyzed to form a test sequence; finally, the robotic arm is guided to complete the replay of the test sequence through visual judgment. On the one hand, the approach in this paper does not access the interior of the app, shielding the association between test actions and device; on the other hand, it captures more abstract test action information instead of simple operation location records and supports more flexible test action replay. We demonstrate the effectiveness of this approach by evaluating the learning-replay of 12 popular apps for 13 typical scenarios on the same device, across devices, and across platforms.

show abstract

“…They use a single camera [14] for collecting data in the form of videos or images and are somewhat immune to shakes. Computer vision [15] is an area of artificial intelligence that can derive meaningful information and data from images or videos. Typically, a camera is placed in front of the vehicle facing the road and the video that is being recorded is analysed and potholes are detected by the AI.…”

Section: Introductionmentioning

confidence: 99%

Enhanced pothole detection system using YOLOX algorithm

K.C

2022

Auton. Intell. Syst.

View full text Add to dashboard Cite

The road is the most commonly used means of transportation and serves as a country’s arteries, so it is extremely important to keep the roads in good condition. Potholes that happen to appear in the road must be repaired to keep the road in good condition. Spotting potholes on the road is difficult, especially in a country like India where roads stretch millions of kilometres across the country. Therefore, there is a need to automate the identification of potholes with high speed and real-time precision. YOLOX is an object detection algorithm and our main goal of this article is to train and analyse the YOLOX model for pothole detection. The YOLOX model is trained with a pothole dataset and the results obtained are analysed by calculating the accuracy, recall and size of the model which is then compared to other YOLO algorithms. The experimental results in this article show that the YOLOX-Nano model predicts potholes with higher accuracy compared to other models while having low computational costs. We were able to achieve an Average Precision (AP) value of 85.6% from training the model and the total size of the model is 7.22 MB. The pothole detection capabilities of the newly developed YOLOX algorithm have never been tested before and this paper is one of the first to detect potholes using the YOLOX object detection algorithm. The research conducted in this paper will help reduce costs and increase the speed of pothole identification and will be of great help in road maintenance.

show abstract

A Survey on the Use of Computer Vision to Improve Software Engineering Tasks

Cited by 11 publications

References 104 publications

Automatically Detecting Visual Bugs in HTML5 Games

Automatically Detecting Visual Bugs in HTML5 Games

Learning-Replay Based Automated Robotic Testing for Mobile App

Enhanced pothole detection system using YOLOX algorithm

Contact Info

Product

Resources

About