Minimum spanning tree dynamic programming stereo‐matching method based on superpixels

This paper presents a novel method for real‐time generation of seamless spherical panoramic videos from an omnidirectional multi‐camera system (OMS). Firstly, a multi‐view video alignment model called spherical projection constrained thin‐plate spline (SP‐TPS) was established and estimated using an approximately symmetrical seam‐line, maintaining the structure inconsistency around the seam‐line. Then, a look‐up table was designed to support real‐time processing of video re‐projection, video dodging and seam‐line updates. In the table, the overlapping areas in OMS multi‐view videos, the seam‐lines between spherical panoramas and OMS multi‐view videos and the pixel coordinate mapping relationship between spherical panoramas and OMS multi‐view videos were pre‐stored as a whole. Finally, a spherical panoramic video was outputted in real‐time through look‐up table computation under an ordinary GPU processor. The experiments were conducted on multi‐view video taken by “1 + 4” and “1 + 7” OMS, respectively. Experimental results demonstrate that compared with four state‐of‐the‐art methods reported in the literature and two bits of commercial software for video stitching, the proposed method excels in eliminating visual artefacts and demonstrates superior adaptability to scenes with varying depths of field. Assuming that OMS is not movable in the scene, this method can generate seamless spherical panoramic videos with a resolution of 8 K in real time, which is of great value to the surveillance field.

Section: Image Stitchingmentioning

confidence: 99%

Real‐time generation of spherical panoramic video using an omnidirectional multi‐camera system

Gao,

Wu,

Huang

et al. 2024

“…In this context, many researchers have exploited the a priori structures inferred from the images or from additional inputs. For instance, planar structures from images (Stathopoulou et al, 2023; Xu & Tao, 2020), an additional digital elevation model in the object space (Ling et al, 2016), a filter window based on surface orientation constraints (Huang & Qin, 2020), similar segments established by a minimum spanning tree (Wang & Xu, 2023; Yang, 2012), a self‐adaptive triangle constraint or line segments in the images (Qin et al, 2019; Ye & Wu, 2018; Zhu et al, 2010) can facilitate the determination of the supporting domain for image matching. The window can also be constructed using non‐symmetrical shifts (Bobick & Intille, 1999) that are meticulously selected by avoiding the cross‐sectional regions of the object contours (Wu et al, 2012), the assumptions of the a priori structures are not universal, and the determination of shiftable windows is non‐trivial.…”

Section: Related Workmentioning

confidence: 99%

Adaptive region aggregation for multi‐view stereo matching using deformable convolutional networks

Hu,

Su,

Mao

et al. 2023

Deep‐learning methods have demonstrated promising performance in multi‐view stereo (MVS) applications. However, it remains challenging to apply a geometrical prior on the adaptive matching windows to achieve efficient three‐dimensional reconstruction. To address this problem, this paper proposes a learnable adaptive region aggregation method based on deformable convolutional networks (DCNs), which is integrated into the feature extraction workflow for MVSNet method that uses coarse‐to‐fine structure. Following the conventional pipeline of MVSNet, a DCN is used to densely estimate and apply transformations in our feature extractor, which is composed of a deformable feature pyramid network (DFPN). Furthermore, we introduce a dedicated offset regulariser to promote the convergence of the learnable offsets of the DCN. The effectiveness of the proposed DFPN is validated through quantitative and qualitative evaluations on the BlendedMVS and Tanks and Temples benchmark datasets within a cross‐dataset evaluation setting.

“…Multi‐view stereo (MVS) is a crucial component in image‐based 3D reconstruction (Seitz et al., 2006; Wang, Wang, et al., 2021; Yao et al., 2018). The plane‐sweep (Collins, 1996) algorithm extends binocular stereo matching (Wang & Xu, 2023) to the task of matching between multiple patches, thereby achieving improved completeness and accuracy of reconstruction by additional shooting angles. Throughout the entire process of 3D reconstruction, structure from motion (SfM; Parente et al., 2019) is initially used to determine the position and orientation of each captured image.…”

Section: Introductionmentioning

confidence: 99%

Mono‐MVS: textureless‐aware multi‐view stereo assisted by monocular prediction

Fu,

Zheng,

Chen

et al. 2024

The learning‐based multi‐view stereo (MVS) methods have made remarkable progress in recent years. However, these methods exhibit limited robustness when faced with occlusion, weak or repetitive texture regions in the image. These factors often lead to holes in the final point cloud model due to excessive pixel‐matching errors. To address these challenges, we propose a novel MVS network assisted by monocular prediction for 3D reconstruction. Our approach combines the strengths of both monocular and multi‐view branches, leveraging the internal semantic information extracted from a single image through monocular prediction, along with the strict geometric relationships between multiple images. Moreover, we adopt a coarse‐to‐fine strategy to gradually reduce the number of assumed depth planes and minimise the interval between them as the resolution of the input images increases during the network iteration. This strategy can achieve a balance between the computational resource consumption and the effectiveness of the model. Experiments on the DTU, Tanks and Temples, and BlendedMVS datasets demonstrate that our method achieves outstanding results, particularly in textureless regions.