Spline Error Weighting for Robust Visual-Inertial Fusion

Abstract

In this paper we derive and test a probability-based weighting that can balance residuals of different types in spline fitting. In contrast to previous formulations, the proposed spline error weighting scheme also incorporates a prediction of the approximation error of the spline fit. We demonstrate the effectiveness of the prediction in a synthetic experiment, and apply it to visual-inertial fusion on rolling shutter cameras. This results in a method that can estimate 3D structure with metric scale on generic first-person videos. We also propose a quality measure for spline fitting, that can be used to automatically select the knot spacing. Experiments verify that the obtained trajectory quality corresponds well with the requested quality. Finally, by linearly scaling the weights, we show that the proposed spline error weighting minimizes the estimation errors on real sequences, in terms of scale and end-point errors.

Trajectory representation and landmark projection for continuous-time structure from motion

“…This work is an extension of the Spline Fusion visual–inertial fusion framework introduced by Lovegrove et al (2013). In this section we outline how the Spline Fusion method works, and also summarize the improvements to robustness of the framework, introduced by Ovrén and Forssén (2018).…”

“…Lovegrove et al (2013), who introduced (5), used a fixed knot spacing value of Δ t = 0 . 1 , and set the norm weight matrices to the inverse covariance of the respective measurement noises. Ovrén and Forssén (2018) showed why these choices are suboptimal, and derived a robust method to set these values. We now give a summary of this method, which is called spline error weighting .…”

“…In this case the residuals in (5) will consist of two error terms: the measurement noise, and a spline approximation error . The Spline Fusion method only accounts for the former, and in Ovrén and Forssén (2018) it is shown that ignoring the approximation error leads to reconstruction failures.…”

“…Here σ r is the empirical residual standard deviation, σ n is the noise standard deviation, which is used by Patron-Perez et al (2015) to predict σ r , Predicted is the spline error weighting residual noise prediction, and σ r 0 is the residual with respect to the noise-free signal x 0 ( t ) . This illustration originally appeared in Ovrén and Forssén (2018). Used here with permission.…”

“…The signals X are based on the accelerometer and gyroscope measurements, because these contain information about both orientation and translation. See Ovrén and Forssén (2018) for further details.…”