Multi-Scale Raft: Combining Hierarchical Concepts for Learning-Based Optical Flow Estimation

Jahedi, Azin; Mehl, Lukas; Rivinius, Marc; Bruhn, Andrés

doi:10.1109/icip46576.2022.9898048

Cited by 11 publications

(6 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Compared to the results of our MS-RAFT method [1], our new MS-RAFT+ approach leads to a significant improvement in accuracy for KITTI 2015 and Sintel (clean). Largest improvements can be observed in non-occluded regions (non-occluded Fl-all, EPE matched) as well as for small displacements (s0-10).…”

Section: Ms-raft+ Vs Ms-raftmentioning

confidence: 85%

“…Our method for the Robust Vision Challenge 2022 is based on our recent MS-RAFT approach, which combines four hierarchical concepts in one estimation framework: (i) a coarse-to-fine computation scheme that provides useful initialization from coarser scales, (ii) a U-Net-style feature extractor which relies on semantically enhanced multiscale features, (iii) RAFT's original correlation pyramid that introduces non-local cost information in the matching process, and (iv) a multi-scale multi-iteration loss that considers a sample-wise robust loss function for fine-tuning the network. We refer the reader to [1] for more details. For the Robust Vision Challenge, we extended the architecture of MS-RAFT by an additional finer scale, which is realized by an on-demand computation of the matching costs, and an adjusted fine-tuning step combining MS-RAFT's sample-wise robust loss function with a different mixed training strategy for better generalization.…”

Section: Approachmentioning

confidence: 99%

“…These feature volumes are computed via our U-Net-style feature encoder introduced in MS-RAFT, which we extended to include an additional finer scale. As shown in [1] considering proper higher resolution inputs for estimating the flow together with a coarse-to-fine strategy improves the accuracy and allows the preservation of more structural details. Therefore, we also extended MS-RAFT's coarse-to-fine scheme to contain an additional finer scale to gain further improvements.…”

Section: Architecturementioning

confidence: 99%

“…Note that unlike RAFT and MS-RAFT, MS-RAFT+ does not pre-compute the all-pairs cost volume due to the extensive memory overhead imposed by the finer scales. Instead, the correlation costs of match candidates are computed on-demand during training and inference 1 . Without the pre-computed all-pairs cost volume, one cannot pre-compute the correlation pyramid.…”

Section: Architecturementioning

confidence: 99%

“…As training supervision we used the multi-scale multi-iteration loss of MS-RAFT. As proposed in [1] we considered the L2 loss for pre-training and the robust sample-wise loss with q = 0.7 for fine-tuning.…”

Section: Trainingmentioning

confidence: 99%

See 4 more Smart Citations

High Resolution Multi-Scale RAFT (Robust Vision Challenge 2022)

Jahedi¹,

Luz²,

Mehl³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this report, we present our optical flow approach, MS-RAFT+, that won the Robust Vision Challenge 2022. It is based on the MS-RAFT method, which successfully integrates several multi-scale concepts into single-scale RAFT. Our approach extends this method by exploiting an additional finer scale for estimating the flow, which is made feasible by on-demand cost computation. This way, it can not only operate at half the original resolution, but also use MS-RAFT's shared convex upsampler to obtain full resolution flow. Moreover, our approach relies on an adjusted fine-tuning scheme during training. This in turn aims at improving the generalization across benchmarks. Among all participating methods in the Robust Vision Challenge, our approach ranks first on VIPER and second on KITTI, Sintel, and Middlebury, resulting in the first place of the overall ranking.

show abstract

Section: Ms-raft+ Vs Ms-raftmentioning

confidence: 85%

Section: Approachmentioning

confidence: 99%

Section: Architecturementioning

confidence: 99%

Section: Architecturementioning

confidence: 99%

Section: Trainingmentioning

confidence: 99%

See 3 more Smart Citations

High Resolution Multi-Scale RAFT (Robust Vision Challenge 2022)

Jahedi¹,

Luz²,

Mehl³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

MS-RAFT+: High Resolution Multi-Scale RAFT

Jahedi,

Luz,

Rivinius

et al. 2023

Int J Comput Vis

View full text Add to dashboard Cite

Hierarchical concepts have proven useful in many classical and learning-based optical flow methods regarding both accuracy and robustness. In this paper we show that such concepts are still useful in the context of recent neural networks that follow RAFT’s paradigm refraining from hierarchical strategies by relying on recurrent updates based on a single-scale all-pairs transform. To this end, we introduce MS-RAFT+: a novel recurrent multi-scale architecture based on RAFT that unifies several successful hierarchical concepts. It employs a coarse-to-fine estimation to enable the use of finer resolutions by useful initializations from coarser scales. Moreover, it relies on RAFT’s correlation pyramid that allows to consider non-local cost information during the matching process. Furthermore, it makes use of advanced multi-scale features that incorporate high-level information from coarser scales. And finally, our method is trained subject to a sample-wise robust multi-scale multi-iteration loss that closely supervises each iteration on each scale, while allowing to discard particularly difficult samples. In combination with an appropriate mixed-dataset training strategy, our method performs favorably. It not only yields highly accurate results on the four major benchmarks (KITTI 2015, MPI Sintel, Middlebury and VIPER), it also allows to achieve these results with a single model and a single parameter setting. Our trained model and code are available at https://github.com/cv-stuttgart/MS_RAFT_plus.

show abstract