Leveraging Graphical User Interface Automation for Generic Robot Programming

Abstract

A novel approach to generic (or generalized) robot programming and a novel simplified, block-based programming environment, called “Assembly”, are introduced. The approach leverages the newest graphical user interface automation tools and techniques to generate programs in various proprietary robot programming environments by emulating user interactions in those environments. The “Assembly” tool is used to generate robot-independent intermediary program models, which are translated into robot-specific programs using a graphical user interface automation toolchain. The generalizability of the approach to list, tree, and block-based programming is assessed using three different robot programming environments, two of which are proprietary. The results of this evaluation suggest that the proposed approach is feasible for an entire range of programming models and thus enables the generation of programs in various proprietary robot programming environments. In educational settings, the automated generation of programs fosters learning different robot programming models by example. For experts, the proposed approach provides a means for generating program (or task) templates, which can be adjusted to the needs of the application at hand on the shop floor.

A Comparative Case Study to Experiences of High School Students Using Text-Based versus Hybrid-Based Environments in Programming Education

“…Trying to catch the logic of coding while combining the blocks correctly requires focusing on multiple parameters. This may make difficult learning (Ionescu, 2021). These contrasting findings may have resulted from based on different programming approaches in the studies.…”

“…On the other hand, in hybrid-based programming, students also need to comprehend the logic of coding, while they combine blocks correctly. This may cause more mental effort and also make it difficult for the learning process (Debue & Van De Leemput, 2014;Ionescu, 2021;Sweller, 2010). Considering these contradictory statements in the literature, it is important to investigate students' experiences in the learning process of a programming language by using Blockly as a hybrid-based programming environment.…”

“…Previous studies are mostly in the engineering field, and on the usability of a technological system developed using Blockly (Adi & Kitagawa, 2019;Bak, Chang, & Choi, 2020;Chen, Chen, Yu, & Lee, 2021;Ionescu, 2021;Jung et al, 2021;Rahaman et al, 2020;Rodríguez-Gil et al, 2019;Weintrop, Shepherd, Francis, & Franklin, 2017;Winterer, Salomon, Köberle, Ramler, & Schittengruber, 2020). There are also available studies using Blockly in programming education at different grade levels.…”

Crowdsourced Evaluation of Robot Programming Environments: Methodology and Application

“…The CoMeMak project aims to democratize cobot technology for a wide variety of users, reaching beyond traditional industrial contexts, such as factories. The framework was assessed by applying it in the evaluation of a simplified robot programming tool, called Assembly, which was developed as part of the CoMeMak project [8,9].…”

Remote User Interface for ABB IRB120 Robot Using TCP/IP Communication

“…In [7] a web application and GUI that acts as a front-end for programming robot motion tasks lowers the engineering complexity. As stated in [8] a more general solution (independent of the robot manufacturer) can be done by using GUI automation tools and generic programming language to generate intermediary programs which are translated into vendor-specific program or template. In both cases certain programming skills and robotics knowledge are still required.…”