Unsupervised Monocular Depth Estimation for Night-Time Images Using Adversarial Domain Feature Adaptation

Vankadari, Madhu; Garg, Sourav; Majumder, Anima; Kumar, Swagat; Behera, Ardhendu

doi:10.1007/978-3-030-58604-1_27

Cited by 41 publications

(40 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the next step, IMUTube calculates the camera ego-motion to account for 3D human pose location and orientation in accordance with the camera movements. IMUTube does so by estimating the background depth maps from each scene and lifting them to 3D point cloud models [62,63]. Based on subsequent 3D point clouds, IMUTube calculates the camera's ego-motion using the Iterative Closest Points (ICP) method [64].…”

Section: D Human Motion Tracking and Virtual Imu Data Extractionmentioning

confidence: 99%

Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables

Kwon

Abowd

Plötz

2021

Sensors

View full text Add to dashboard Cite

Supervised training of human activity recognition (HAR) systems based on body-worn inertial measurement units (IMUs) is often constrained by the typically rather small amounts of labeled sample data. Systems like IMUTube have been introduced that employ cross-modality transfer approaches to convert videos of activities of interest into virtual IMU data. We demonstrate for the first time how such large-scale virtual IMU datasets can be used to train HAR systems that are substantially more complex than the state-of-the-art. Complexity is thereby represented by the number of model parameters that can be trained robustly. Our models contain components that are dedicated to capture the essentials of IMU data as they are of relevance for activity recognition, which increased the number of trainable parameters by a factor of 1100 compared to state-of-the-art model architectures. We evaluate the new model architecture on the challenging task of analyzing free-weight gym exercises, specifically on classifying 13 dumbbell execises. We have collected around 41 h of virtual IMU data using IMUTube from exercise videos available from YouTube. The proposed model is trained with the large amount of virtual IMU data and calibrated with a mere 36 min of real IMU data. The trained model was evaluated on a real IMU dataset and we demonstrate the substantial performance improvements of 20% absolute F1 score compared to the state-of-the-art convolutional models in HAR.

show abstract

Section: D Human Motion Tracking and Virtual Imu Data Extractionmentioning

confidence: 99%

Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables

Kwon

Abowd

Plötz

2021

Sensors

View full text Add to dashboard Cite

show abstract

“…In the past two years, more and more scholars have used adversarial discriminant learning methods to predict depth information with better results. [41] train an encoder that extracts imperceptible nighttime features that distinguish daytime images by an adversarial discriminative learning method based on PatchGAN, and plug a pre-trained daytime depth decoder into its back end to achieve unsupervised nighttime monocular depth estimation. S3Net [42] considers the geometric structure across space and time in monocular video frames in an adversarial network framework, i.e., using geometric, temporal, and semantic constraints simultaneously for depth prediction.…”

Section: B Self-supervised Monocular Depth Estimationmentioning

confidence: 99%

Mixed-Scale Unet Based on Dense Atrous Pyramid for Monocular Depth Estimation

et al. 2021

View full text Add to dashboard Cite

Monocular depth estimation is an undirected problem, so constructing a network to predict better image depth information is an important research topic. This paper proposes a mixed-scale Unet network (MAPNet) with a dense atrous pyramid based on the coder-decoder structure widely used in computer vision. We innovatively introduce the Unet++ structure of the image segmentation network for depth estimation. We reset the number of convolutional layers of the network under the framework of the Unet++ network and innovatively connect the decoders densely. Moreover, by choosing the appropriate size of the atrous radius, we form a dense atrous pyramid based on different feature layers to better connect the features in the deep and shallow layers of the network. To verify the effectiveness of the proposed algorithm, we tested the network on the KITTI public dataset and the NYU Depth V2 dataset. We compared it with the current state-of-the-art methods. The proposed method has higher accuracy and has steadily improved relative to the threshold of accuracy and root-mean-square error. We also conduct ablation studies, studies targeting the effectiveness of the network framework, and discussions on the convergence time and parameter complexity of the network. We will open-source the code at https://github.com/yang-yi-fan/MAPUnet.

show abstract

“…To estimate all day time depth, [19] utilizes a thermal imaging camera sensor to reduce the influence of low-visibility in the night-time, while [26] adds LiDAR to provide additional information in estimating depth maps at night-time. Meanwhile, using generate adversarial network, [33] and [34] propose effective strategies for depth estimation of night-time images. [33] utilizes a translation network with light effects and uninformative regions that can render realistic night stereo images from day stereo images, and vice versa.…”

Section: Night-time Depth Estimationmentioning

confidence: 99%

Self-supervised Monocular Depth Estimation for All Day Images using Domain Separation

Liu¹,

Song²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Remarkable results have been achieved by DCNN based self-supervised depth estimation approaches. However, most of these approaches can only handle either day-time or night-time images, while their performance degrades for all-day images due to large domain shift and the variation of illumination between day and night images. To relieve these limitations, we propose a domain-separated network for self-supervised depth estimation of all-day images. Specifically, to relieve the negative influence of disturbing terms (illumination, etc.), we partition the information of day and night image pairs into two complementary sub-spaces: private and invariant domains, where the former contains the unique information (illumination, etc.) of day and night images and the latter contains essential shared information (texture, etc.). Meanwhile, to guarantee that the day and night images contain the same information, the domain-separated network takes the day-time images and corresponding night-time images (generated by GAN) as input, and the private and invariant feature extractors are learned by orthogonality and similarity loss, where the domain gap can be alleviated, thus better depth maps can be expected. Meanwhile, the reconstruction and photometric losses are utilized to estimate complementary information and depth maps effectively. Experimental results demonstrate that our approach achieves state-of-theart depth estimation results for all-day images on the challenging Oxford RobotCar dataset, proving the superiority of our proposed approach. Code and data split are available at https://github.com/LINA-lln/ADDS-DepthNet.

show abstract

Unsupervised Monocular Depth Estimation for Night-Time Images Using Adversarial Domain Feature Adaptation

Cited by 41 publications

References 39 publications

Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables

Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables

Mixed-Scale Unet Based on Dense Atrous Pyramid for Monocular Depth Estimation

Self-supervised Monocular Depth Estimation for All Day Images using Domain Separation

Contact Info

Product

Resources

About