Real-Time Panoptic Segmentation From Dense Detections

Hou, Rui; Li, Jie; Bhargava, Arjun; Raventos, Allan; Guizilini, Vitor; Fang, Chao; Lynch, Jerome P.; Gaidon, Adrien

doi:10.1109/cvpr42600.2020.00855

Cited by 66 publications

(38 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although our segmentation quality for stuff classes is lower (27.1 vs. 28.3), our panoptic segmentation head with soft attention masks brings a significant improvement for things categories, where PQ th is 4.5 points higher. The single-stage DenseBox [11] outperforms our results with a 2.7 PQ margin, at the cost of higher inference time (63 ms vs. 45 ms). We observe that our weakness lies in the segmentation quality for stuff categories.…”

Section: Performance On Cocomentioning

confidence: 54%

“…With a more powerful backbone, VoVNet2-39-FPNlite, we match the accuracy of Panoptic DeepLab. From the single stage methods, our network achieves the best results in terms of both accuracy, surpassing FPSNet [10] with 4.2 PQ and DenseBox [11] with 0.5 PQ. In terms of speed, our method is slightly outperformed by Prototype Panoptic [28] but is more accurate, with a PQ score difference of 2.…”

Section: Performance On Cityscapesmentioning

confidence: 98%

“…Motivated by their benefits, FPSNet [10] builds upon RetinaNet [17] and achieves a fast inference speed. DenseBox [11] leverages the idea of reusing discarded dense object proposals by matching them with final object bounding boxes. Ref.…”

Section: Related Workmentioning

confidence: 99%

“…Most recent approaches [6,7] focus on achieving high accuracy on public benchmarks by designing novel semantic segmentation heads on top of Mask R-CNN [8] instance segmentation network or by designing clever merging heuristics within the network [9]. However, significant progress has been achieved by fast networks, starting from [10], which builds upon a one-stage detector, to [11], where dense bounding boxes are clustered into instance masks using semantic segmentation and finally, the proposal-free method [12], which predicts instance centers and regresses instance center offsets.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Fast Panoptic Segmentation with Soft Attention Embeddings

Petrovai

Nedevschi

2022

Sensors

View full text Add to dashboard Cite

Panoptic segmentation provides a rich 2D environment representation by unifying semantic and instance segmentation. Most current state-of-the-art panoptic segmentation methods are built upon two-stage detectors and are not suitable for real-time applications, such as automated driving, due to their high computational complexity. In this work, we introduce a novel, fast and accurate single-stage panoptic segmentation network that employs a shared feature extraction backbone and three network heads for object detection, semantic segmentation, instance-level attention masks. Guided by object detections, our new panoptic segmentation head learns instance specific soft attention masks based on spatial embeddings. The semantic masks for stuff classes and soft instance masks for things classes are pixel-wise coherent and can be easily integrated in a panoptic output. The training and inference pipelines are simplified and no post-processing of the panoptic output is necessary. Benefiting from fast inference speed, the network can be deployed in automated vehicles or robotic applications. We perform extensive experiments on COCO and Cityscapes datasets and obtain competitive results in both accuracy and time. On the Cityscapes dataset we achieve 59.7 panoptic quality with an inference speed of more than 10 FPS on high resolution 1024 × 2048 images.

show abstract

Section: Performance On Cocomentioning

confidence: 54%

Section: Performance On Cityscapesmentioning

confidence: 98%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Fast Panoptic Segmentation with Soft Attention Embeddings

Petrovai

Nedevschi

2022

Sensors

View full text Add to dashboard Cite

show abstract

“…And as can be seen in table III, using a smaller images increases inference speed by 2× at the expense of 3 Panoptic Quality points. PQ, SQ and RQ correspond to the Panoptic Quality, Architecture PQ(%) Time(ms) JSISNet [17] 17.6 n/a AUNet [5] 59.0 n/a Panoptic FPN [3] 58.1 n/a Single Network PS [18] 42.9 590 DeeperLab [8] 56.53 308 Panoptic Deeplab [9] 63.0 175 AdapIS [10] 62.0 n/a FPSNet [19] 55.1 114 Real-Time PS [20] 58…”

Section: A Image Size and Inference Timementioning

confidence: 99%

YOLO-based Panoptic Segmentation Network

Diaz-Zapata

Erkent

Laugier

2021

2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC)

View full text Add to dashboard Cite

Autonomous vehicles need information about their surroundings to safely navigate them. For this, the task of Panoptic Segmentation is proposed as a method of fully parsing the scene by assigning each pixel a label and instance id. Given the constraints of autonomous driving, this process needs to be done in a fast manner. In this paper, we propose the first panoptic segmentation network based on the YOLOv3 real-time object detection network by adding a semantic and instance segmentation branches. YOLO-panoptic is able to do real-time inference and achieves a performance similar to the state of the art methods in some metrics.

show abstract

Developing an expansion‐based obstacle detection using panoptic segmentation

Pirasteh,

Varshosaz,

Badrloo

et al. 2024

Journal of Field Robotics

View full text Add to dashboard Cite

Safe Micro Aerial Vehicle (MAV) navigation requires detecting and avoiding obstacles. For safe MAV navigation, expansion‐based algorithms are effective for detecting obstacles. However, accurate and real‐time obstacle detection is a fundamental challenge. Some traditional methods focus on extracting geometric features from images and applying geometric constraints to identify potential obstacles. Others may leverage machine learning algorithms for object detection and classification, using features such as texture, shape, and context to distinguish obstacles from background clutter. The choice of approach depends on factors such as the specific requirements of the application, the complexity of the scene, and the available computational resources. Since obstacles, in reality, take the form of objects (e.g., persons, walls, pillars, trees, automobiles, and other structures), it is preferable to represent them according to human comprehension and as objects. Therefore, the objective of this study is to reflect on the previous research and address the issues mentioned above by extracting objects from the fisheye image using a panoptic deep‐learning network. The extracted object regions are, then, used to identify obstacles with a novel area‐based expansion rate we developed in a previous study. We compared the accuracy of obstacle detection in our proposed method to the existing method when moving forward and to the right; thus, we improved it between 10% and 18%, respectively. In addition, compared with the existing method, and due to replacing a single object with multiple regions, obstacle‐detection runtime for forward and right direction is 15.71 and 25.5 times faster, respectively, and the required match points have decreased by 49% and 55%.

show abstract

Real-Time Panoptic Segmentation From Dense Detections

Cited by 66 publications

References 25 publications

Fast Panoptic Segmentation with Soft Attention Embeddings

Fast Panoptic Segmentation with Soft Attention Embeddings

YOLO-based Panoptic Segmentation Network

Developing an expansion‐based obstacle detection using panoptic segmentation

Contact Info

Product

Resources

About