Domain Generalization via Optical Flow: Training a CNN in a Low-Quality Simulation to Detect Obstacles in the Real World

Learning to Predict Collision Risk from Simulated Video Data

“…Other works have successfully leveraged optical flow [12] or disparity maps [13] to significantly reduce the sim-toreal performance gap. However, the applicability of both works is limited, as they evaluate relatively simple scenarios where collisions with (often static) objects within only a few meters are predicted.…”

“…FlowDroNet [12] and Xie et al [13] have successfully shown good generalization from simulated to real-world data by using optical flow or depth as a task-specific input representation. FlowDroNet is closest to our work as it includes non-static objects and supervised learning, therefore we focus on comparing to their work and leave depth as intermediate modality for future work.…”

“…FlowDroNet is closest to our work as it includes non-static objects and supervised learning, therefore we focus on comparing to their work and leave depth as intermediate modality for future work. Flow inherently generalizes better to unseen data compared to camera video data due to its decreased visual complexity, as textures, colors and illuminations do not exist in optical flow [12]. Additionally, optical flow represents motion which is inherently relevant to collision prediction.…”

“…Collision prediction studies have successfully directed their models' attention by taking a crop of each vehicle detection and extracting a set of local features from the crops [14], [15]. However, as our features are extracted from optical flow rather than RGB, significantly fewer features can be extracted from local objects [12]. In other classification problems, research has shown that fusing attention masks in feature space rather than taking crops in the data-space shows increased performance [25].…”

“…As flow consists of a horizontal and vertical component, two H ×W images are provided at every t. Since optical flow is estimated based on two observed frames, we require T + 1 RGB images for T optical flow frames. We experiment with training on ground truth optical flow provided in D src (with pre-processing similarly to [12]) and training on flow estimated by LiteFlowNet [22] on the simulated camera data. Fig.…”