Event-Oriented 3D Convolutional Features Selection and Hash Codes Generation Using PCA for Video Retrieval

Abstract

The extensive video surveillance networks gather an enormous amount of data exponentially on a daily basis and its management is a challenging task, requiring efficient and effective techniques for searching, indexing, and retrieval. The employed mainstream techniques are focusing on general category videos, where the important events in surveillance require fine-grained events retrieval. In this paper, we introduce an event-oriented feature selection mechanism by utilizing the intermediate convolutional layer of a pre-trained 3D-CNN model, that is selected after deep investigation of its weights and response to a particular event. The extracted exclusive features represent an event semantically and effectively eliminate those neurons which do not respond to that event. Furthermore, the event-oriented convolutional features are of very high-dimensions, requiring additional storage, and take more time in feature comparison for retrieval. Therefore, we generate compact binary codes from these features using principle component analysis (PCA) algorithm. This makes our system more efficient to retrieve videos from large scale database. We evaluated our approach on the challenging events of UCF101 and HMDB51 datasets for original features and generated compact codes, achieving faster execution time, and better precision and recall scores. INDEX TERMS Deep learning, feature selection, video retrieval, video analytics, hash codes, surveillance event analysis.

Content-Based Video Retrieval With Prototypes of Deep Features

“…Thus, recent research applied 3D CNNs that represent video data including temporal information more efficiently. Ullah et al [17] used a pre-trained C3D model [18] on the Sports-1M dataset [24] and reduced the dimensionality by means of PCA to generate the hash code. However, according to work by Hara et al [19], Kataoka et al [20], and Tran et al [29], working with the Sports-1M dataset is not easy because it contains more videos than the Kinetics 400 and Kinetics 700 datasets [30], [31], and their annotations are noisier than those of these Kinetics datasets.…”

“…In other words, the ActivityNet dataset has much noisier frames which are not closely related to the video context compared to the UCF101 and HMDB51 datasets. Table 5 shows the mAP results of the proposed PCA-CBVR with the R3D50 feature extractor, which shows the best performance as shown in Figure 5 Methods UCF101 HMDB51 ActivtyNet Event-Oriented Video Retrieval [17] 0.83 0.75 -Stacked HetConv-MK-BiDLSTM [16] --0.17 PCA-CBVR(without fine-tuning) 0.81 0.57 0.35…”

“…Finally, as a result of the fine search, it returns the top-k videos most similar to the user's query video. [14] of the sampled frames using traditional computer vision techniques or deep features [15]- [17] of the sampled frames using a convolutional neural network (CNN). The last step is the distance calculation step which compares extracted features from both the database videos and the query video in terms of the Euclidean distance or cosine similarity metric.…”

Fuzzy Logic in Surveillance Big Video Data Analysis

“…Our first limitation is that the targeted domain for reviewed papers in this survey is comparatively narrow, i.e., surveillance BVD. A broader version can be the complete video analytics domain, covering action and activity recognition [60], video summarization, [61], video retrieval [7], healthcare [62], objects detection and tracking [63,64], etc. The specific focus of our research is to conduct an in-depth review of fuzzy methods applied to the generic video analysis domain, towards deriving a proper taxonomy of the applied fuzzy techniques.…”

“…As such, the trained models show limited performance in complex smart cities surveillance applications [6]. For improved utilization of surveillance BVD, researchers are investigating several image processing and machine learning algorithms [7].…”