Video analytics using deep learning for crowd analysis: a review

Abstract

Gathering a large number of people in a shared physical area is very common in urban culture. Although there are limitless examples of mega crowds, the Islamic religious ritual, the Hajj, is considered as one of the greatest crowd scenarios in the world. The Hajj is carried out once in a year with a congregation of millions of people when the Muslims visit the holy city of Makkah at a given time and date. Such a big crowd is always prone to public safety issues, and therefore requires proper measures to ensure safe and comfortable arrangement. Through the advances in computer vision based scene understanding, automatic analysis of crowd scenes is gaining popularity. However, existing crowd analysis algorithms might not be able to correctly interpret the video content in the context of the Hajj. This is because the Hajj is a unique congregation of millions of people crowded in a small area, which can overwhelm the use of existing video and computer vision based sophisticated algorithms. Through our studies on crowd analysis, crowd counting, density estimation, and the Hajj crowd behavior, we faced the need of a review work to get a research direction for abnormal behavior analysis of Hajj pilgrims. Therefore, this review aims to summarize the research works relevant to the broader field of video analytics using deep learning with a special focus on the visual surveillance in the Hajj. The review identifies the challenges and leading-edge techniques of visual surveillance in general, which may gracefully be adaptable to the applications of Hajj and Umrah. The paper presents detailed reviews on existing techniques and approaches employed for crowd analysis from crowd videos, specifically the techniques that use deep learning in detecting abnormal behavior. These observations give us the impetus to undertake a painstaking yet exhilarating journey on crowd analysis, classification and detection of any abnormal movement of the Hajj pilgrims. Furthermore, because the Hajj pilgrimage is the most crowded domain for video-related extensive research activities, this study motivates us to critically analyze the crowd on a large scale.

Joint Solution for Temporal-Spatial Synchronization of Multi-View Videos and Pedestrian Matching in Crowd Scenes

“…For crowd data extraction, existing studies have primarily focused on positioning in the imaging plane [1], with business objectives centred around crowd behaviour analysis, crowd counting, and density estimation [55,56]. Due to misidentifications caused by obstructions and high density in crowd scenarios, and combined with the three problems C1~C3, the complexity has been raised, and no research currently addresses all three problems simultaneously.…”

Deep Learning in Diverse Intelligent Sensor Based Systems

“…Video analytics refers to generating descriptions of the content of, or events in the video, which involves tasks of object (persons, cars, or other objects) detection, tracking, as well as calculating their appearance and movements. It is also an important and essential computer vision technique and has significant practical benefits such as monitoring video for security incidents helps prevent crime, intelligent traffic systems, and more [ 270 , 271 ]. While its tasks overlap beyond image analysis tasks, they are more challenging because they involve both spatial and temporal information.…”

“…Other representative RCNN based models include the one proposed by Ballas et al [ 273 ], MaskRNN [ 274 ], and MoNet [ 275 ]. For comprehensive discussions of video analytics we refer to recent surveys [ 270 , 271 , 276 , 277 ].…”

Efficient single- and multi-DNN inference using TensorRT framework

“…Nowadays, artificial intelligence (AI) video analytics systems are used extensively in multiple real-life applications, including self-driving cars [1,2], surveillance [3], medicine [4], document recognition [5] and agriculture [6]. In the pursuit of high recognition accuracy, the core components of these systems -deep neural networks (DNNs) -have become more and more computationally complex [7].…”