On hierarchical modelling of motion for workflow analysis from overhead view

Abstract

Understanding human behaviour is a high level perceptual problem, one which is often dominated by the contextual knowledge of the environment, and where concerns such as occlusion, scene clutter and high within-class variations are commonplace. Nonetheless, such understanding is highly desirable for automated visual surveillance. We consider this problem in a context of a workflow analysis within an industrial environment. The hierarchical nature of the workflow is exploited to split the problem into 'activity' and 'task' recognition. In this, sequences of low level activities are examined for instances of a task while the remainder are labelled as background. An initial prediction of activity is obtained using shape and motion based features of the moving blob of interest. A sequence of these activities is further adjusted by a probabilistic analysis of transitions between activities using hidden Markov models (HMMs). In task detection, HMMs are arranged to handle the activities within each task. Two separate HMMs for task and background compete for an incoming sequence of activities. Imagery derived from a camera mounted overhead the target scene has been chosen over the more conventional oblique views (from the side) as this view does not suffer from as much occlusion, and it poses a manageable detection and tracking problem while still retaining powerful cues as to the workflow patterns. We evaluate our approach both in activity and task detection on a challenging dataset of surveillance of human operators in a car manufacturing plant. The experimental results show that our hierarchical approach can automatically segment the timeline and spatially localize a series of predefined tasks that are performed to complete a workflow.

Improving Time Study Methods Using Deep Learning-Based Action Segmentation Models

“…In the context of manufacturing, a limited number of papers were found, with notable works being [6,7,[33][34][35][36][37] The application of classic machine learning models was predominant in these papers, where models like Hidden Markov Models [6,34,35] or Support Vector Machines [7] were employed following the manual extraction of features. Makantasis et al [33] applied a deep learning model based on a 2D convolutional neural network and multi-layer perceptron, using manually created features with the Motion History Image algorithm.…”

“…In paper [37], a system comprising three stages was developed: spatial feature extraction using a Vectors Assembly Graph (VAG) and graph networks from RGB-D video frames; contact force feature extraction via a sliding window technique; and action segmentation through a multi-stage temporal convolution network (MS-TCN) that combines these features. Jiang et al [7] collected data in laboratory conditions, while studies [33][34][35] were conducted using a dataset described in [38], which is no longer publicly available. Rude et al [6] used a dataset from [39], collected with a depth sensor for painting manufactured parts, recording data over a single workday with two different workers.…”

Modeling Human Motion for Predicting Usage of Hospital Operating Room

“…The more general problem of workflow monitoring is already being addressed in more constrained industrial environments such as car manufacturing (Voulodimos et al, 2011;Veres et al, 2011;Arbab-Zavar et al, 2014). In 2014, Arbab-Zavar et al (Arbab-Zavar et al, 2014) exploited shape and motion features extracted from an overhead video in order to identify highly structured tasks and activities within a car manufacturing plant. A Markov temporal structure based decision system has been proposed in (Behera et al, 2014) to model spatio-temporal relationships during object manipulations tasks and has been tested for continuous activity recognition in assembling a pump system.…”

Joint Abnormal Blob Detection and Localization Under Complex Scenes

“…State event models were described in [8][9][10]. Bayesian network event model utilizes probability as a mechanism for handling the uncertainty of observations and interpreting existing events in a video.…”