GPU-FV: Realtime Fisher Vector and Its Applications in Video Monitoring

Abstract

Fisher vector has been widely used in many multimedia retrieval and visual recognition applications with good performance. However, the computation complexity prevents its usage in real-time video monitoring. In this work, we proposed and implemented GPU-FV, a fast Fisher vector extraction method with the help of modern GPUs. The challenge of implementing Fisher vector on GPUs lies in the data dependency in feature extraction and expensive memory access in Fisher vector computing. To handle these challenges, we carefully designed GPU-FV in a way that utilizes the computing power of GPU as much as possible, and applied optimizations such as loop tiling to boost the performance. GPU-FV is about 12 times faster than the CPU version, and 50% faster than a non-optimized GPU implementation. For standard video input (320*240), GPU-FV can process each frame within 34ms on a model GPU. Our experiments show that GPU-FV obtains a similar recognition accuracy as traditional FV on VOC 2007 and Caltech 256 image sets. We also applied GPU-FV for realtime video monitoring tasks and found that GPU-FV outperforms a number of previous works. Especially, when the number of training examples are small, GPU-FV outperforms the recent popular deep CNN features borrowed from ImageNet.

Fast MPEG-CDVS Encoder With GPU-CPU Hybrid Computing

“…Wang et al further analyzed the workload of SIFT in [16] and proposed to distribute the feature extraction tasks to CPU and GPU, such that a speed of 10 fps for a 320 × 256 image and 41% energy consumption reduction can be achieved. Besides SIFT, the speeded-up robust feature (SURF) [3], [18] and fisher vectors (FV) [19] were also explored in implementing using GPU platform, and around an order of magnitude speedup was achieved compared to CPU based implementation. Besides hand-crafted features, recently convolutional neural network based features have achieved promising performance in various computer vision tasks such as image classification [20] and retrieval [21].…”