Camera-IMU (Inertial Measurement Unit) sensor fusion has been extensively studied in recent decades. Numerous observability analysis and fusion schemes for motion estimation with self-calibration have been presented. However, it has been uncertain whether both camera and IMU intrinsic parameters are observable under general motion. To answer this question, we first prove that for a global shutter camera-IMU system, all intrinsic and extrinsic parameters are observable with an unknown landmark. Given this, time offset and readout time of a rolling shutter (RS) camera also prove to be observable. Next, to validate this analysis and to solve the drift issue of a structureless filter during standstills, we develop a Keyframebased Sliding Window Filter (KSWF) for odometry and selfcalibration, which works with a monocular RS camera or stereo RS cameras. Though the keyframe concept is widely used in vision-based sensor fusion, to our knowledge, KSWF is the first of its kind to support self-calibration. Our simulation and real data tests validated that it is possible to fully calibrate the camera-IMU system using observations of opportunistic landmarks under diverse motion. Real data tests confirmed previous allusions that keeping landmarks in the state vector can remedy the drift in standstill, and showed that the keyframe-based scheme is an alternative cure.

Nonlinear systems of affine control inputs overarch many sensor fusion instances. Analyzing whether a state variable in such a nonlinear system can be estimated (i.e., observability) informs better estimator design. Among the research on local observability of nonlinear systems, approaches based on differential geometry have attracted much attention for the solid theoretic foundation and suitability to automated deduction. Such approaches usually work with a system model of unconstrained control inputs and assume that the control inputs and observation outputs are timestamped by the same clock. To our knowledge, it has not been shown how to conduct the observability analysis with additional constraints enforced on the system's observations or control inputs. To this end, we propose procedures to convert a system model of affine control inputs with linear constraints into a constraint-free standard model which is apt to be analyzed by the classic observability analysis procedure. Then, the whole analysis procedure is illustrated by applying to the well-studied visual inertial odometry (VIO) system which estimates the camera-IMU relative pose and time offset. The findings about unobservable variables under degenerate motion concur with those obtained with linearized VIO systems in other studies, whereas the findings about observability of time offset extend those in previous studies. These findings are further validated by simulation.

The rolling shutter (RS) mechanism is widely used by consumer-grade cameras, which are essential parts in smartphones and autonomous vehicles. The RS effect leads to image distortion upon relative motion between a camera and the scene. This effect needs to be considered in video stabilization, structure from motion, and vision-aided odometry, for which recent studies have improved earlier global shutter (GS) methods by accounting for the RS effect. However, it is still unclear how the RS affects spatiotemporal calibration of the camera in a sensor assembly, which is crucial to good performance in aforementioned applications.This work takes the camera-IMU system as an example and looks into the RS effect on its spatiotemporal calibration. To this end, we develop a calibration method for a RScamera-IMU system with continuous-time B-splines by using a calibration target. Unlike in calibrating GS cameras, every observation of a landmark on the target has a unique camera pose fitted by continuous-time B-splines. With simulated data generated from four sets of public calibration data, we show that RS can noticeably affect the extrinsic parameters, causing errors about 1 • in orientation and 2 cm in translation with a RS setting as in common smartphone cameras. With real data collected by two industrial camera-IMU systems, we find that considering the RS effect gives more accurate and consistent spatiotemporal calibration. Moreover, our method also accurately calibrates the inter-line delay of the RS. The code for simulation and calibration is publicly available 1 .