Learning to Estimate Hidden Motions with Global Motion Aggregation

Jiang, Shihao; Campbell, Dylan; Lu, Yao; Li, Hongdong; Hartley, Richard

doi:10.1109/iccv48922.2021.00963

Cited by 254 publications

(140 citation statements)

References 31 publications

Supporting

Mentioning

140

Contrasting

Order By: Relevance

“…In particular, RAFT [71] showed the effectiveness of an all-pairs inter-frame correlation volume as an encoding, which is essentially an attention map. All-pairs intra-frame correlations were subsequently shown to help resolve correspondence ambiguities [34]. For longer-range correspondences, object tracking by repeated detection [58] and data association can be used.…”

Section: Related Workmentioning

confidence: 99%

Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers

Patrick¹,

Campbell²,

Asano³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In video transformers, the time dimension is often treated in the same way as the two spatial dimensions. However, in a scene where objects or the camera may move, a physical point imaged at one location in frame t may be entirely unrelated to what is found at that location in frame t + k. These temporal correspondences should be modeled to facilitate learning about dynamic scenes. To this end, we propose a new drop-in block for video transformers-trajectory attention-that aggregates information along implicitly determined motion paths. We additionally propose a new method to address the quadratic dependence of computation and memory on the input size, which is particularly important for high resolution or long videos. While these ideas are useful in a range of settings, we apply them to the specific task of video action recognition with a transformer model and obtain state-of-the-art results on the Kinetics, Something-Something V2, and Epic-Kitchens datasets. Code and models are available at: https://github.com/facebookresearch/ Motionformer. * Equal contribution.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers

Patrick¹,

Campbell²,

Asano³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…During training, we follow prior works (Teed and Deng 2020;Jiang et al 2021a) to adopt AdamW optimizer with one-cycle learning rate policy, and conduct model pretraining on synthetic data as the standard optical flow training procedure. The model is pretrained on FlyingChairs (Dosovitskiy et al 2015) for 180k iterations and then on FlyingThings (Mayer et al 2016) for 180k iterations.…”

Section: Implementation Detailsmentioning

confidence: 99%

Learning Optical Flow with Adaptive Graph Reasoning

Luo¹,

Yang²,

Luo³

et al. 2022

Preprint

View full text Add to dashboard Cite

Estimating per-pixel motion between video frames, known as optical flow, is a long-standing problem in video understanding and analysis. Most contemporary optical flow techniques largely focus on addressing the cross-image matching with feature similarity, with few methods considering how to explicitly reason over the given scene for achieving a holistic motion understanding. In this work, taking a fresh perspective, we introduce a novel graph-based approach, called adaptive graph reasoning for optical flow (AGFlow), to emphasize the value of scene/context information in optical flow.Our key idea is to decouple the context reasoning from the matching procedure, and exploit scene information to effectively assist motion estimation by learning to reason over the adaptive graph. The proposed AGFlow can effectively exploit the context information and incorporate it within the matching procedure, producing more robust and accurate results. On both Sintel clean and final passes, our AGFlow achieves the best accuracy with EPE of 1.43 and 2.47 pixels, outperforming state-of-the-art approaches by 11.2% and 13.6%, respectively. Codes will be publicly available at https: //github.com/LA30/AGFlow.

show abstract

“…The performance of EV-FlowNet [53] and E-RAFT is taken from Gehrig et al [14]. We additionally train the frame-based RAFT model [41] and GMA [21], an addition over RAFT, to also compare against purely frame-based approaches.…”

Section: Dsec-flowmentioning

confidence: 99%

“…On top of this, estimating pixel motion in the wild must deal with occlusions and objects leaving the camera field of view. Modern optical flow methods [21] attempt to address partially occluded objects but still completely fail if an independent object is only visible in one of the two frames (see Figure 6).…”

Section: Introductionmentioning

confidence: 99%

Dense Continuous-Time Optical Flow from Events and Frames

Gehrig¹,

Muglikar²,

Scaramuzza³

2022

Preprint

View full text Add to dashboard Cite

We present a method for estimating dense continuous-time optical flow. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. In this work, we show that it is possible to compute perpixel, continuous-time optical flow by additionally using events from an event camera. Events provide temporally fine-grained information about movement in image space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous-time via parameterized Bézier curves. To achieve this, we introduce multiple innovations to build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. The proposed approach outperforms existing image-based and event-based methods by 11.5 % lower EPE on DSEC-Flow. Finally, we introduce a novel synthetic dataset MultiFlow for pixel trajectory regression on which our method is currently the only successful approach.

show abstract

Learning to Estimate Hidden Motions with Global Motion Aggregation

Cited by 254 publications

References 31 publications

Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers

Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers

Learning Optical Flow with Adaptive Graph Reasoning

Dense Continuous-Time Optical Flow from Events and Frames

Contact Info

Product

Resources

About