Self-Supervised Camera Self-Calibration from Video

Jiading, Fang,; Igor, Vasiljevic,; Guizilini, Vitor; Ambruş, Rareş; Shakhnarovich, Greg; Gaidon, Adrien; Walter, Matthew R.

doi:10.1109/icra46639.2022.9811784

Cited by 15 publications

(8 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The HR‐Depth only includes a depth estimation network and a pose network, so it assumes that camera intrinsic parameters are known, which makes it impossible to train a self‐supervised model from arbitrary video sequences. Inspired by the self‐calibration method of a camera with a unified camera model (Fang et al., 2022), a camera network in the SMDEU is proposed to estimate camera intrinsic parameters in Figure 1. Thus, the SMDEU can simultaneously estimate monocular depth and learn camera pose and intrinsics from input video sequences.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Monocular 3D object detection for construction scene analysis

Shen,

Jiao,

Zhang

et al. 2023

Computer aided Civil Eng

View full text Add to dashboard Cite

Three‐dimensional (3D) object detection, that is, localizing and classifying all critical objects in a 3D space, is essential for downstream construction scene analysis tasks. However, accurate instance segmentation, few 2D object segmentation and 3D object detection data sets, high‐quality feature representations for depth estimation, and limited 3D cues from a single red‐green‐blue (RGB) image pose significant challenges to 3D object detection and severely hinder its practical applications. In response to these challenges, an improved cascade‐based network with a transformer backbone and a boundary‐patch‐refinement method is proposed to build hierarchical features and refine object boundaries, resulting in better results in 2D object detection and instance segmentation. Furthermore, a novel self‐supervised monocular depth learning method is proposed to extract better feature representations for depth estimation from construction site video data with unknown camera parameters. Additionally, a pseudo‐LiDAR point cloud method and a 3D object detection method with a density‐based clustering algorithm are proposed to detect 3D objects in a construction scene without help from 3D labels, which will serve as a good foundation for other downstream 3D tasks. Finally, the proposed model is evaluated for object instance segmentation and depth estimation on the moving objects in construction sites (MOCS) and construction scene data sets. It brings a 9.16% gain in terms of mean average precision (mAP) for object detection and a 4.92% gain in mask mAP for object instance segmentation. The average order accuracy and relative mean error for depth estimation are improved by 0.94% and 60.56%, respectively. This study aims to overcome the challenges and limitations of 3D object detection and facilitate practical applications in construction scene analysis.

show abstract

Section: Methodsmentioning

confidence: 99%

“…Inspired by the self-calibration method of a camera with a unified camera model (Fang et al, 2022), a camera network in the SMDEU is proposed to estimate camera intrinsic parameters in Figure 1. Thus, the SMDEU can simultaneously estimate monocular depth and learn camera pose and intrinsics from input video sequences.…”

Section: 2mentioning

confidence: 99%

Monocular 3D object detection for construction scene analysis

Shen,

Jiao,

Zhang

et al. 2023

Computer aided Civil Eng

View full text Add to dashboard Cite

show abstract

“…The total degree freedom of camera parameters is 7(𝑡 has 2 degrees of freedom because the scale cannot be recovered, 𝑅 has three degrees of freedom, and each camera has 1 degree of which is equal to the fundamental matrix 𝐹's degree of freedom. The internal and external parameters of the image can be recovered using a self-calibration method [37]. But even if these constraints are met, The solved camera parameters by [37] suffers from large errors when the scene is approximately planar and the matching error is large.…”

Section: Estimation Of Image Intrinsic and Extrinsic Parametersmentioning

confidence: 99%

“…The internal and external parameters of the image can be recovered using a self-calibration method [37]. But even if these constraints are met, The solved camera parameters by [37] suffers from large errors when the scene is approximately planar and the matching error is large. Therefore, here we use the method of optimizing the objective function in [6] to solve the internal and external parameters of the camera.…”

Section: Estimation Of Image Intrinsic and Extrinsic Parametersmentioning

confidence: 99%

SC-AOF: A Sliding Camera and Asymmetric Optical Flow-Based Blending Method for Image Stitching

Chang,

Li,

Liang

et al. 2024

Preprint

View full text Add to dashboard Cite

Parallax processing and structure preservation has long been important and challenging tasks in image stitching. In this paper, an image stitching method based on sliding camera to eliminate perspective deformation and asymmetric optical flow to solve parallax is proposed. By maintaining the viewpoint of two input images in the mosaic non-overlapping area and creating a virtual camera by interpolation in the overlapping area, the viewpoint is gradually transformed from one to another image view-point so as to complete the smooth transition of the two image viewpoints and reduce perspective deformation. Two coarsely aligned warped images are generated with the help of a global projection plane. After that, the optical flow propagation and gradient descent method are used to quickly calculate the bidirectional asymmetric optical flow between the two warped images, and the optical flow-based method is used to further align the two warped images to reduce parallax. In the image blending, the softmax function and registration error are used to adjust the width of the blending area, further eliminating ghosting and reduce parallax. Finally, the comparative experiments show that our method can effectively eliminate perspective deformation, reduce misalignment, and save running time.

show abstract

“…Substantial work has been done to develop and demonstrate the methods for extracting range data from monocular RGB images 4 – 6 Universally, these methods use convolutional neural networks (CNNs) to identify objects and determine their range. Since camera images are the source data, the locations in the scene are determined by geometric calculations based on the position of the detected objects on the sensor.…”

Section: Monocular Thermal Rangingmentioning

confidence: 99%

Monocular thermal imaging for pedestrian detection and ranging

Petilli¹

2022

Opt. Eng.

View full text Add to dashboard Cite

To react to the presence of pedestrians, an automated braking system must first find the pedestrian(s) including locations relative to the automobile both in angular position and in distance. In the daytime, cameras and radar can provide the necessary information, but this combination, which requires ambient or active illumination, fails at night. Passive thermal sensors are now being enlisted to dramatically improve imaging at night, whereas substantial effort is underway to assure proper fusing of object information from the thermal sensor with other sensors on the automobile. To simplify the acquisition of information needed to make valid automated braking decisions at night, a camera system with a thermal image sensor was developed that identifies pedestrians and labels each identified pedestrian with location and distance data. The camera utilizes a single uncooled custom microbolometer sensor and a software suite implementing artificial intelligence and machine learning capabilities, running on a combination of convolutional neural networks and fusion processing to provide the data that an automobile host computer needs to implement fast, safe, and accurate automatic braking. We present details of the system construction and operation as well as initial test results showing the potential this technology has to dramatically reduce pedestrian fatalities at night as well as augment safety across all conditions, whether day, night, fog, rain, snow, dust, sun glare, or headlight glare.

show abstract

Self-Supervised Camera Self-Calibration from Video

Cited by 15 publications

References 32 publications

Monocular 3D object detection for construction scene analysis

Monocular 3D object detection for construction scene analysis

SC-AOF: A Sliding Camera and Asymmetric Optical Flow-Based Blending Method for Image Stitching

Monocular thermal imaging for pedestrian detection and ranging

Contact Info

Product

Resources

About