Stereoscopic image stitching with rectangular boundaries

Abstract

This paper proposes a novel algorithm for stereoscopic image stitching, which aims to produce stereoscopic panoramas with rectangular boundaries. As a result, it provides wider field of view and better viewing experience for users. To achieve this, we formulate stereoscopic image stitching and boundary rectangling in a global optimization framework that simultaneously handles feature alignment, disparity consistency and boundary regularity. Given two (or more) stereoscopic images with overlapping content, each containing two views (for left and right eyes), we represent each view using a mesh and our algorithm contains three main steps: We first perform a global optimization to stitch all the left views and right views simultaneously, which ensures feature alignment and disparity consistency. Then, with the optimized vertices in each view, we extract the irregular boundary in the stereoscopic panorama, by performing polygon Boolean operations in left and right views, and construct the rectangular boundary constraints. Finally, through a global energy optimization, we warp left and right views according to feature alignment, disparity consistency and rectangular boundary constraints. To show the effectiveness of our method, we further extend our method to disparity adjustment and stereoscopic stitching with large horizon. Experimental results show that our method can produce visually pleasing stereoscopic panoramas without noticeable distortion or visual fatigue, thus resulting in satisfactory 3D viewing experience.

Rectangling irregular videos by optimal spatio-temporal warping

“…This idea has been widely used in many applications like image resizing [34], perspective editing [2,35], etc. Image warping can also be used for rectangling images with irregular boundaries [15], which provides the effect of image completion, and has been extended to process stereoscopic panoramas [7,16]. Actually, the method of Ref.…”

“…Actually, the method of Ref. [7] claims to be the first to deal with rectangling stereoscopic stitched images, while the method of Ref. [16] extends it to stereoscopic panoramic video.…”

“…In recent years, geometry techniques have been widely used in image and video processing, such as image resizing or retargetting [1], perspective editing [2,3], video stabilization [4], and panoramic stitching [5][6][7][8][9]. Unlike traditional image and video processing based on pixel operations, geometry-driven methods typically adopt a mesh grid structure in the image plane and manipulate the grid points to drive the processing of the enclosed patches, enabling more flexible and coherent processing of the image and video content [10].…”

“…Such a method can well fill the gaps along the irregular boundaries by warping the whole image, with content preservation; it is able to generate more natural transitions than image completion methods. This rectangling strategy has also been adopted for processing stereoscopic panoramas [7,16], which has fewer requirements for temporal consistency involving feature correspondence and motion. In this paper, we further extend this method to irregular videos by rectangling frames in a spatially and temporally coherent manner.…”

VR content creation and exploration with deep learning: A survey

“…The raw images and video are then warped and stitched to make a 360 • panorama [31] for VR display. The normal pipeline in conventional panorama stitching techniques [3,[32][33][34] consists of 2D transformation estimation and seamless stitching with blending [35]. However, they cannot produce acceptable results if correctly matched feature points are lacking.…”