Detecting Anomalies from Human Activities by an Autonomous Mobile Robot based on “Fast and Slow” Thinking

Region Anomaly Detection via Spatial and Semantic Attributed Graph in Human Monitoring

“…On the other hand, the latter methods explicitly combine the images or regions with their relationships as the contexts to understand and discover diverse image-level or region-level anomalies, such as video surveillance [ 1 , 2 ] and human monitoring [ 7 , 8 , 9 ]. Among such works, several approaches [ 9 , 15 , 16 , 27 , 28 ] consider the regions and their relations in the visual perspective for region anomaly detection, while our previous methods [ 7 , 8 ] additionally adopt deep-captioning models, such as DenseCap [ 17 ], to obtain region captions as the semantic information for the task. Sun et al [ 27 ] proposed a Spatio-Temporal Graph (STG) to represent spatio-temporal relations among objects to bridge the gap between an anomaly and its context.…”

“…Among such works, image region anomaly detection [ 7 , 8 , 9 , 10 , 11 ] is a vital task of spotting abnormal areas from images in human monitoring. Traditional methods focus on discovering region-level anomalies that deviate from the patterns learned from normal image regions [ 7 , 10 , 11 , 12 , 13 ].…”

“…For instance, a man holding a baseball bat in the laboratory [ 7 ] is a single anomaly, as such behavior is never observed in normal regions. However, in addition to the single anomalies, there also exist contextual anomalies [ 8 , 9 , 14 ], which violate regular interactions among human and objects, as the context of a region is characterized by other regions in the same image. For instance, the region of a man making a phone call is normal when it is located close to a kitchen sink and a soap bottle, as they are in a resting area, while abnormal when close to a bookshelf and a notebook PC, as they are in a working area if the latter is not allowed.…”

“…Existing methods consider region contexts by exploring the relations among regions. They can be classified into an object-label-based method [ 15 ], a spatial-relation-based method [ 16 ], and deep-captioning-based methods [ 7 , 8 ]. Choi et al [ 15 ] represent all the objects in an image with a tree-structured model to detect objects that do not conform to the scene.…”

“…The spatial-relation-based method [ 16 ] considers the positions, such as above, below, and inside, of two objects to detect abnormal semantic relationships between a pair of image segmentations, while such spatial positions are limited in characterizing diverse region contexts that are essential for detecting the contextual anomalies. In addition to exploiting visual features of image regions, our previous methods [ 7 , 8 ] adopt deep-captioning models, such as DenseCap [ 17 ], to obtain region captions as the semantic information for the task. Since these methods also consider both the visual and semantic information of image regions on the same task, they are the most relevant works to our proposed method.…”