A Supervised Video Hashing Method Based on a Deep 3D Convolutional Neural Network for Large-Scale Video Retrieval

Abstract

Recently, with the popularization of camera tools such as mobile phones and the rise of various short video platforms, a lot of videos are being uploaded to the Internet at all times, for which a video retrieval system with fast retrieval speed and high precision is very necessary. Therefore, content-based video retrieval (CBVR) has aroused the interest of many researchers. A typical CBVR system mainly contains the following two essential parts: video feature extraction and similarity comparison. Feature extraction of video is very challenging, previous video retrieval methods are mostly based on extracting features from single video frames, while resulting the loss of temporal information in the videos. Hashing methods are extensively used in multimedia information retrieval due to its retrieval efficiency, but most of them are currently only applied to image retrieval. In order to solve these problems in video retrieval, we build an end-to-end framework called deep supervised video hashing (DSVH), which employs a 3D convolutional neural network (CNN) to obtain spatial-temporal features of videos, then train a set of hash functions by supervised hashing to transfer the video features into binary space and get the compact binary codes of videos. Finally, we use triplet loss for network training. We conduct a lot of experiments on three public video datasets UCF-101, JHMDB and HMDB-51, and the results show that the proposed method has advantages over many state-of-the-art video retrieval methods. Compared with the DVH method, the mAP value of UCF-101 dataset is improved by 9.3%, and the minimum improvement on JHMDB dataset is also increased by 0.3%. At the same time, we also demonstrate the stability of the algorithm in the HMDB-51 dataset.

Deep Video Hashing Using 3DCNN with BERT

“…The comparison for the UCF101 and HMDB51 is illustrated in Table 1. Values of other compared methods are taken from [31,37,42] or the referenced papers directly. The metric compared being the mean average precision (mAP).…”

“…You'll find the least variance in mAP in our model because of separating the feature learning and hash learning process. And minor variance in [42]. This is due to having the model train on feature extraction first, then the model was modified to incorporate hash learning as fine-tuning of the model.…”

“…They are separate in the training phase to be able to fine-tune each learning processes individually. And to decrease the variance in the result between each hash code size as can be noticed in [42]. Other works that relied on the model learning to hash end to end through hashing losses show minor to noticeable variance between the results [34,37].…”

“…HMDB51 is used to evaluate performance of human action recognition DH [41] 0.300 0.290 0.470 ---SRH [30] 0.716 0.692 0.754 0.781 --DVH [32] 0.701 0.705 0.712 ---KSH [6] --0.716 0.786 0.810 0.848 DBH [33] --0.681 0.736 0.766 0.785 DNNH [34] --0.740 0.789 0.817 0.835 SVHM [42] --0.798 0.801 0.806 -CSQ [37] 0.838 0.875 0.874 ---DHWCM [31] -0. DH [41] 0.360 0.360 0.310 ---SRH [30] 0.491 0.503 0.509 ---DVH [32] 0.441 0.456 0.518 ---KSH [6] --0.431 0.464 0.450 0.473 DBH [33] --0.389 0.391 0.386 0.346 DNNH [34] --0.487 0.503 0.493 0.480 CSQ [37] 0.527 0.565 0.579 ---SVHM [42] --0.562 0.565 0.575 -DHWCM [31] -0.487 0.605 0.588 0.588 0.672 R(2+1) BERT (Proposed) 0.848 0.849 0.854 0.852 0.849 0.849 system. The actions in this dataset are divided into 5 groups: General facial actions, general body movements, facial actions, body movements and body movement for human interaction.…”

An efficient approach for large-scale image-to-video retrieval with convolutional neural network features

“…There are two main components in a typical content-based video retrieval (CBVR) system, one is video feature extraction, and another is similarity comparison. Extracting video features is very challenging; previous video feature extraction methods were primarily based on individual video frames, leading to the loss of temporal information in videos [30].…”