Understanding 3D Semantic Structure around the Vehicle with Monocular Cameras

Narioka, Kenichi; Nishimura, Hiroki; Itamochi, Takayuki; Inomata, Teppei

doi:10.1109/ivs.2018.8500397

Cited by 14 publications

(5 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Modern scene segmentation approaches are mostly designed to work with pinhole images on mainstream datasets like Cityscapes [19] and Mapillary Vistas [20]. To enlarge the Field of View (FoV), early surrounding perception systems are based on fisheye images or multiple cameras [4] [21]. Motivated by the prospect of attaining wide-angle semantic perception with a single camera, recent works [22][23] build directly on this modality, relying on synthetic collections that are far less diverse than pinhole databases [19] [20].…”

Section: B Panoramic Segmentationmentioning

confidence: 99%

“…This additional information about the surrounding is critical for many real-world applications such as autonomous driving and various other robotic navigation tasks. In particular, Intelligent Vehicles (IV) require an indepth understanding of the surrounding world [4].…”

Section: Introductionmentioning

confidence: 99%

“…A field of view beyond classical pinhole images is crucial for any dynamic environment, otherwise objects of interest can move out of sight [3]. Using multiple cameras or cameras along with other sensors such as LiDAR could be one solution to overcome this problem [4] [5]. However, it requires multiple different systems to work in parallel and combine those information to a holistic picture [5], which incurs higher latency and computation complexity that are clearly not advantageous for intelligent vehicle applications.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning

Jaus

Yang

Stiefelhagen

2021

2021 IEEE Intelligent Vehicles Symposium (IV)

View full text Add to dashboard Cite

In this work, we introduce panoramic panoptic segmentation as the most holistic scene understanding both in terms of field of view and image level understanding. A complete surrounding understanding provides a maximum of information to the agent, which is essential for any intelligent vehicle in order to make informed decisions in a safetycritical dynamic environment such as real-world traffic. In order to overcome the lack of annotated panoramic images, we propose a framework which allows model training on standard pinhole images and transfers the learned features to a different domain. Using our proposed method, we manage to achieve significant improvements of over 5% measured in PQ over nonadapted models on our Wild Panoramic Panoptic Segmentation (WildPPS) dataset. We show that our proposed Panoramic Robust Feature (PRF) framework is not only suitable to improve performance on panoramic images but can be beneficial whenever model training and deployment are executed on data taken from different distributions. As an additional contribution, we publish WildPPS: The first panoramic panoptic image dataset to foster progress in surrounding perception.

show abstract

Section: B Panoramic Segmentationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning

Jaus

Yang

Stiefelhagen

2021

2021 IEEE Intelligent Vehicles Symposium (IV)

View full text Add to dashboard Cite

show abstract

“…Omnidirectional semantic segmentation. As systems progressed towards 360 • perception, early omnidirectional segmentation works were based on fisheye images [12,74] or focused on indoor scenes [54,56] with surround-view systems in street-scenes being largely based on a multitude of cameras [11,37,41,47]. Motivated by the prospect of attaining 360 • semantic perception based on a singular panoramic camera, recent works build semantic segmentation systems directly on top of this sensormodality [2,68,85].…”

Section: Related Workmentioning

confidence: 99%

Capturing Omni-Range Context for Omnidirectional Segmentation

Yang¹,

Zhang²,

Reiß³

et al. 2021

Preprint

View full text Add to dashboard Cite

Convolutional Networks (ConvNets) excel at semantic segmentation and have become a vital component for perception in autonomous driving. Enabling an allencompassing view of street-scenes, omnidirectional cameras present themselves as a perfect fit in such systems. Most segmentation models for parsing urban environments operate on common, narrow Field of View (FoV) images. Transferring these models from the domain they were designed for to 360 • perception, their performance drops dramatically, e.g., by an absolute 30.0% (mIoU) on established test-beds. To bridge the gap in terms of FoV and structural distribution between the imaging domains, we introduce Efficient Concurrent Attention Networks (ECANets), directly capturing the inherent long-range dependencies in omnidirectional imagery. In addition to the learned attentionbased contextual priors that can stretch across 360 • images, we upgrade model training by leveraging multi-source and omni-supervised learning, taking advantage of both: Densely labeled and unlabeled data originating from multiple datasets. To foster progress in panoramic image segmentation, we put forward and extensively evaluate models on Wild PAnoramic Semantic Segmentation (WildPASS), a dataset designed to capture diverse scenes from all around the globe. Our novel model, training regimen and multisource prediction fusion elevate the performance (mIoU) to new state-of-the-art results on the public PASS (60.2%) and the fresh WildPASS (69.0%) benchmarks. 1

show abstract

“…However, they sacrificed partial vertical FoV for the purpose of preserving straight lines when segmenting unwarpped images on cylindrical projection surfaces. Analogously, K. Narioka et al 19 equipped a vehicle with five cameras equiangularly for a wider FoV to perceive surroundings. They designed a set of efficient Deep Neural Networks (DNNs) and trained only with front facing camera images.…”

Section: Related Workmentioning

confidence: 99%

Semantic segmentation of panoramic images using a synthetic dataset

Wang

Yang

et al. 2019

Artificial Intelligence and Machine Learning in Defense Applications

View full text Add to dashboard Cite

Panoramic images have advantages in information capacity and scene stability due to their large field of view (FoV). In this paper, we propose a method to synthesize a new dataset of panoramic image. We managed to stitch the images taken from different directions into panoramic images, together with their labeled images, to yield the panoramic semantic segmentation dataset denominated as SYNTHIA-PANO. For the purpose of finding out the effect of using panoramic images as training dataset, we designed and performed a comprehensive set of experiments. Experimental results show that using panoramic images as training data is beneficial to the segmentation result. In addition, it has been shown that by using panoramic images with a 180 degree FoV as training data the model has better performance. Furthermore, the model trained with panoramic images also has a better capacity to resist the image distortion. Our codes and SYNTHIA-PANO dataset are available: https://github.com/Francis515/SYNTHIA-PANO

show abstract

Understanding 3D Semantic Structure around the Vehicle with Monocular Cameras

Cited by 14 publications

References 8 publications

Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning

Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning

Capturing Omni-Range Context for Omnidirectional Segmentation

Semantic segmentation of panoramic images using a synthetic dataset

Contact Info

Product

Resources

About