Deep learning-based visual control assistant for assembly in Industry 4.0

Zamora-Hernández, Mauricio-Andrés; Castro-Vargas, John Alejandro; Azorín-López, Jorge; García-Rodríguez, José

doi:10.1016/j.compind.2021.103485

Cited by 38 publications

(12 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Zheng et al presented in [13] an augmented-reality-based assistant system for aircraft cable assembly with a convolutional neural network for deep vision that provides rapid guidance, reduces errors, and mitigates the dependency on hard copy documents. Another deep learning visual assistant for assembly tasks in production, recognizing real-time tools and worker assembly actions to reduce rework and waste, is presented in [1]. The proposed solution relies on a generic description language that was developed to characterize the actions within an assembly process.…”

Section: Related Workmentioning

confidence: 99%

“…To compete successfully in the global market, in recent years, factories have turned their attention to optimize all tasks, including the ones performed by humans, leveraging the progress in information technology with the deployment of artificial intelligence [1,2] and machine learning [3] in various application areas throughout the product life cycle. Industry 4.0 [4] is the coined term used to describe this optimization involving interconnection and collaboration among the factory's interactants (human and synthetic) towards a human-automation symbiosis.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Robust Assembly Assistance Using Informed Tree Search with Markov Chains

Gellért

Sorostinean

Pirvu

2022

Sensors

View full text Add to dashboard Cite

Manual work accounts for one of the largest workgroups in the European manufacturing sector, and improving the training capacity, quality, and speed brings significant competitive benefits to companies. In this context, this paper presents an informed tree search on top of a Markov chain that suggests possible next assembly steps as a key component of an innovative assembly training station for manual operations. The goal of the next step suggestions is to provide support to inexperienced workers or to assist experienced workers by providing choices for the next assembly step in an automated manner without the involvement of a human trainer on site. Data stemming from 179 experiment participants, 111 factory workers, and 68 students, were used to evaluate different prediction methods. From our analysis, Markov chains fail in new scenarios and, therefore, by using an informed tree search to predict the possible next assembly step in such situations, the prediction capability of the hybrid algorithm increases significantly while providing robust solutions to unseen scenarios. The proposed method proved to be the most efficient for next assembly step prediction among all the evaluated predictors and, thus, the most suitable method for an adaptive assembly support system such as for manual operations in industry.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Robust Assembly Assistance Using Informed Tree Search with Markov Chains

Gellért

Sorostinean

Pirvu

2022

Sensors

View full text Add to dashboard Cite

show abstract

“…Moreover, the recent wide-spread of low-cost video camera systems, including depth-cameras 6 , has strengthened the development of observation systems in a variety of application domains such as video-surveillance, safety and smart home security, ambient assisted living, health-care and so on. However, little work has been done in human action recognition for manufacturing assembly 7 – 9 and the poor availability of public datasets limits the study, development, and comparison of new methods. This is mainly due to challenging issues such as between-action similarity, the complexity of actions, the manipulation of tools and parts, the presence of fine motions and intricate operations.…”

Section: Background and Summarymentioning

confidence: 99%

“…The recognition of human actions in the context of intelligent manufacturing is of great importance for various purposes: to improve operational efficiency 8 ; to promote human-robot cooperation 10 ; to assist operators 11 ; to support employee training 9 , 12 ; to increase productivity and safety 13 ; or to promote workers’ good mental health 14 . In this paper, we present the Human Action Multi-Modal Monitoring in Manufacturing (HA4M) dataset which is a multi-modal dataset acquired by an RGB-D camera during the assembly of an Epicyclic Gear Train (EGT) (see Fig.…”

Section: Background and Summarymentioning

confidence: 99%

The HA4M dataset: Multi-Modal Monitoring of an assembly task for Human Action recognition in Manufacturing

et al. 2022

View full text Add to dashboard Cite

This paper introduces the Human Action Multi-Modal Monitoring in Manufacturing (HA4M) dataset, a collection of multi-modal data relative to actions performed by different subjects building an Epicyclic Gear Train (EGT). In particular, 41 subjects executed several trials of the assembly task, which consists of 12 actions. Data were collected in a laboratory scenario using a Microsoft® Azure Kinect which integrates a depth camera, an RGB camera, and InfraRed (IR) emitters. To the best of authors’ knowledge, the HA4M dataset is the first multi-modal dataset about an assembly task containing six types of data: RGB images, Depth maps, IR images, RGB-to-Depth-Aligned images, Point Clouds and Skeleton data. These data represent a good foundation to develop and test advanced action recognition systems in several fields, including Computer Vision and Machine Learning, and application domains such as smart manufacturing and human-robot collaboration.

show abstract

“…The use of machine learning and, specifically, deep learning techniques is becoming increasingly common, both in the improvement of industrial production processes and in their preventive maintenance. We find examples such as deep-learningbased visual control assembly assistant [1]. This work enables real-time evaluation of the activities in the assembly process to identify errors.…”

Section: Introductionmentioning

confidence: 99%

Anomaly detection and virtual reality visualisation in supercomputers

Pérez

Benavent-Lledo²,

Azorín-López³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Anomaly detection is the identification of events or observations that deviate from the expected behaviour of a given set of data. Its main application is the prediction of possible technical failures. In particular, anomaly detection on supercomputers is a difficult problem to solve due to the large scale of the systems and the large number of components. Most research works in this field employ machine learning methods and regression models in a supervised fashion, which implies the need for a large amount of labelled data to train such systems. This work proposes the use of autoencoder models, allowing the problem to be approached with semi-supervised learning techniques. Two different model training approaches are compared. The former is a model trained with data from all the nodes of a supercomputer. In the latter approach, observing significant differences between nodes, one model is trained for each node. The results are analysed by evaluating the positive and negative aspects of each approach. On the other hand, a replica of the Marconi 100 supercomputer is developed in a virtual reality environment that allows the data from each node to be visualised at the same time.

show abstract

Deep learning-based visual control assistant for assembly in Industry 4.0

Cited by 38 publications

References 35 publications

Robust Assembly Assistance Using Informed Tree Search with Markov Chains

Robust Assembly Assistance Using Informed Tree Search with Markov Chains

The HA4M dataset: Multi-Modal Monitoring of an assembly task for Human Action recognition in Manufacturing

Anomaly detection and virtual reality visualisation in supercomputers

Contact Info

Product

Resources

About