STAT: Spatial-Temporal Attention Mechanism for Video Captioning

Yan, Chenggang; Tu, Yunbin; Wang, Xingzheng; Zhang, Yongbing; Hao, Xinhong; Zhang, Yongdong; Dai, Qionghai

doi:10.1109/tmm.2019.2924576

Cited by 314 publications

(129 citation statements)

References 48 publications

Supporting

Mentioning

129

Contrasting

Order By: Relevance

“…Descriptors encode important information about the visual characteristics of the objects present in images [34], such as appearance [35,36], motion [37], or geometry [38]. Therefore, they have been used in multiple contexts.…”

Section: Descriptorsmentioning

confidence: 99%

“…Alioua et al [35] propose a 2D head pose estimation framework using a combination of classic descriptors, e.g., HoG, SURF, and Haar. Yan et al [36] uses two CNN features to model global and local appearance of the target and a 3D CNN which codify the motion. The computational cost of some descriptors could be expensive, e.g., especially those based on deep learning [36], even using parallelization methods [37].…”

Section: Descriptorsmentioning

confidence: 99%

“…Yan et al [36] uses two CNN features to model global and local appearance of the target and a 3D CNN which codify the motion. The computational cost of some descriptors could be expensive, e.g., especially those based on deep learning [36], even using parallelization methods [37]. Therefore, we rely on robust features with fair computational cost.…”

Section: Descriptorsmentioning

confidence: 99%

See 2 more Smart Citations

Robust head pose estimation based on key frames for human-machine interaction

Madrigal

Lerasle

2020

J Image Video Proc.

View full text Add to dashboard Cite

Humans can interact with several kinds of machine (motor vehicle, robots, among others) in different ways. One way is through his/her head pose. In this work, we propose a head pose estimation framework that combines 2D and 3D cues using the concept of key frames (KFs). KFs are a set of frames learned automatically offline that consist the following: 2D features, encoded through Speeded Up Robust Feature (SURF) descriptors; 3D information, captured by Fast Point Feature Histogram (FPFH) descriptors; and target's head orientation (pose) in real-world coordinates, which is represented through a 3D facial model. Then, the KF information is re-enforced through a global optimization process that minimizes error in a way similar to bundle adjustment. The KF allows to formulate, in an online process, a hypothesis of the head pose in new images that is then refined through an optimization process, performed by the iterative closest point (ICP) algorithm. This KF-based framework can handle partial occlusions and extreme rotations even with noisy depth data, improving the accuracy of pose estimation and detection rate. We evaluate the proposal using two public benchmarks in the state of the art: (1) BIWI Kinect Head Pose Database and (2) ICT 3D HeadPose Database. In addition, we evaluate this framework with a small but challenging dataset of our own authorship where the targets perform more complex behaviors than those in the aforementioned public datasets. We show how our approach outperforms relevant state-of-the-art proposals on all these datasets.

show abstract

Section: Descriptorsmentioning

confidence: 99%

Section: Descriptorsmentioning

confidence: 99%

Section: Descriptorsmentioning

confidence: 99%

See 1 more Smart Citation

Robust head pose estimation based on key frames for human-machine interaction

Madrigal

Lerasle

2020

J Image Video Proc.

View full text Add to dashboard Cite

show abstract

“…Other than this, the multi-view is also well known as multiple angles cameras of 3D model [16]. For heterogeneous multimedia features such as multi-feature hashing of video data with local and global visual features [17], [18] and other orthogonal or associated features, researchers use multi/cross-modal hashing to solve the complexity of the fusion problem of multiple modalities. It is disparate from the multi-view hashing.…”

Section: Introductionmentioning

confidence: 99%

Deep Multi-View Enhancement Hashing for Image Retrieval

Yan

Gong

Wei

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

350

View full text Add to dashboard Cite

Hashing is an efficient method for nearest neighbor search in large-scale data space by embedding high-dimensional feature descriptors into a similarity preserving Hamming space with a low dimension. However, large-scale high-speed retrieval through binary code has a certain degree of reduction in retrieval accuracy compared to traditional retrieval methods. We have noticed that multi-view methods can well preserve the diverse characteristics of data. Therefore, we try to introduce the multi-view deep neural network into the hash learning field, and design an efficient and innovative retrieval model, which has achieved a significant improvement in retrieval performance. In this paper, we propose a supervised multi-view hash model which can enhance the multi-view information through neural networks. This is a completely new hash learning method that combines multi-view and deep learning methods. The proposed method utilizes an effective view stability evaluation method to actively explore the relationship among views, which will affect the optimization direction of the entire network. We have also designed a variety of multi-data fusion methods in the Hamming space to preserve the advantages of both convolution and multi-view. In order to avoid excessive computing resources on the enhancement procedure during retrieval, we set up a separate structure called memory network which participates in training together. The proposed method is systematically evaluated on the CIFAR-10, NUS-WIDE and MS-COCO datasets, and the results show that our method significantly outperforms the state-of-the-art single-view and multi-view hashing methods.

show abstract

“…The escalation of high specification computers, high-resolution reasonable cameras, and highly dependent video analysis-based applications drive research in object tracking. To date, object tracking is pertinent to the tasks of motion-based recognition, automated surveillance [1], video captioning [2], humancomputer interaction [3], traffic monitoring [4], and autonomous vehicles [5,6].…”

Section: Introductionmentioning

confidence: 99%

Visual Object Tracking Fusing CNN and Color Histogram based Tracker and Depth Estimation for Automatic Immersive Audio Mixing

Park¹,

Islam²,

Baek³

2020

KSII TIIS

View full text Add to dashboard Cite

We propose a robust visual object tracking algorithm fusing a convolutional neural network tracker trained offline from a large number of video repositories and a color histogram based tracker to track objects for mixing immersive audio. Our algorithm addresses the problem of occlusion and large movements of the CNN based GOTURN generic object tracker. The key idea is the offline training of a binary classifier with the color histogram similarity values estimated via both trackers used in this method to opt appropriate tracker for target tracking and update both trackers with the predicted bounding box position of the target to continue tracking. Furthermore, a histogram similarity constraint is applied before updating the trackers to maximize the tracking accuracy. Finally, we compute the depth(z) of the target object by one of the prominent unsupervised monocular depth estimation algorithms to ensure the necessary 3D position of the tracked object to mix the immersive audio into that object. Our proposed algorithm demonstrates about 2% improved accuracy over the outperforming GOTURN algorithm in the existing VOT2014 tracking benchmark. Additionally, our tracker also works well to track multiple objects utilizing the concept of single object tracker but no demonstrations on any MOT benchmark.

show abstract

STAT: Spatial-Temporal Attention Mechanism for Video Captioning

Cited by 314 publications

References 48 publications

Robust head pose estimation based on key frames for human-machine interaction

Robust head pose estimation based on key frames for human-machine interaction

Deep Multi-View Enhancement Hashing for Image Retrieval

Visual Object Tracking Fusing CNN and Color Histogram based Tracker and Depth Estimation for Automatic Immersive Audio Mixing

Contact Info

Product

Resources

About