Worker’s Behavior Identification from Manufacturing Site Images using Mask R-CNN and MediaPipe

Rijayanti, Rita; Hwang, Min‐Shiang; Jin, Kyo-Hong

doi:10.6109/jkiice.2023.27.2.281

Cited by 1 publication

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Subsequently, the collected data were divided into training data of 75% and testing data of 25%. In this particular context, the model was intended to enhance the previous model [15,16] by adding a dataset and layers. For a more detailed explanation, Figure 3a depicts the model specifically detecting each object in the frames: a worker, a machine, a toolbox, and a tool; Figure 3b depicts a worker, a machine, a control box, and two toolboxes displayed using bbox and segmentation masks.…”

Section: Implementation Resultsmentioning

confidence: 99%

“…A support vector machine classifier is implemented to classify human behavior. Furthermore, in [15,16], image analysis is performed on images where objects and body poses remain stable.…”

Section: Related Workmentioning

confidence: 99%

“…In addition, LSTM is utilized to train the model to recognize possible differences in worker behavior within manufacturing environments. Additionally, as an advancement of previous research [15,16], this study focuses on using videos to generate descriptive texts and distinguish normal and anomalous worker behavior.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Detection of Anomalous Behavior of Manufacturing Workers Using Deep Learning-Based Recognition of Human–Object Interaction

2023

View full text Add to dashboard Cite

The increasing demand for industrial products has expanded production quantities, leading to negative effects on product quality, worker productivity, and safety during working hours. Therefore, monitoring the conditions in manufacturing environments, particularly human workers, is crucial. Accordingly, this study presents a model that detects workers’ anomalous behavior in manufacturing environments. The objective is to determine worker movements, postures, and interactions with surrounding objects based on human–object interactions using a Mask R-CNN, MediaPipe Holistic, a long short-term memory (LSTM), and worker behavior description algorithm. The process begins by recognizing the objects within video frames using a Mask R-CNN. Afterward, worker poses are recognized and classified based on object positions using a deep learning-based approach. Next, we identified the patterns or characteristics that signified normal or anomalous behavior. In this case, anomalous behavior consists of anomalies correlated with human pose recognition (emergencies: worker falls, slips, or becomes ill) and human pose recognition with object positions (tool breakage and machine failure). The findings suggest that the model successfully distinguished anomalous behavior and attained the highest pose recognition accuracy (approximately 96%) for standing, touching, and holding, and the lowest accuracy (approximately 88%) for sitting. In addition, the model achieved an object detection accuracy of approximately 97%.

show abstract

Section: Implementation Resultsmentioning

confidence: 99%

“…A support vector machine classifier is implemented to classify human behavior. Furthermore, in [15,16], image analysis is performed on images where objects and body poses remain stable.…”

Section: Related Workmentioning

confidence: 99%