Commands for Autonomous Vehicles by Progressively Stacking Visual-Linguistic Representations

Dai, Hang; Luo, Shujie; Ding, Yong; Shao, Ling

doi:10.1007/978-3-030-66096-3_2

Cited by 13 publications

(18 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The CMTR method [17] applies the Transformer encoder-decoder model to model the command and regions separately. Dai et al [5] use the VL-BERT pre-trained model [18] to jointly learn cross-modal representations for the input. They propose an iterative stacking algorithm, Stack-VL-BERT, to train a deeper VL-BERT model.…”

Section: A Language Grounding For Autonomous Vehiclesmentioning

confidence: 99%

“…We apply the UNITER [4] model from single-stream architecture and LXMERT [10] from dual-stream architecture because they achieve excellent performance in many V&L tasks [38]. On the other hand, SOTA methods [5] in this task uses another single-stream model, VL-BERT [18]. Their model obtains lower scores than UNITER and LXMERT (see Table I for the results).…”

Section: Our Layer Fusion Approachmentioning

confidence: 99%

“…Existing work typically uses an object detector to extract region proposals from the input image and casts the task as selecting the best-matched region based on the command. State-of-the-art (SOTA) methods [5] apply a visionand-language (V&L) pre-trained model to summarize the region proposals and command words into contextualized representations through multiple Transformer [6] encoder layers, and then use the top layer representations to predict Fig. 1: A sample image and the associated natural language command.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attention

Chan¹,

Guo²,

Xu³

2022

Preprint

View full text Add to dashboard Cite

Grounding a command to the visual environment is an essential ingredient for interactions between autonomous vehicles and humans. In this work, we study the problem of language grounding for autonomous vehicles, which aims to localize a region in a visual scene according to a natural language command from a passenger. Prior work only employs the top layer representations of a vision-and-language pretrained model to predict the region referred to by the command. However, such a method omits the useful features encoded in other layers, and thus results in inadequate understanding of the input scene and command. To tackle this limitation, we present the first layer fusion approach for this task. Since different visual regions may require distinct types of features to disambiguate them from each other, we further propose the region-specific dynamic (RSD) layer attention to adaptively fuse the multimodal information across layers for each region. Extensive experiments on the Talk2Car benchmark demonstrate that our approach helps predict more accurate regions and outperforms state-of-the-art methods.

show abstract

Section: A Language Grounding For Autonomous Vehiclesmentioning

confidence: 99%

Section: Our Layer Fusion Approachmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attention

Chan¹,

Guo²,

Xu³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Three dimensional object detector localizes objects with tight 3D bounding boxes. Compared with monocular and stereo image based approaches [5,19,26,27,41], LiDAR based methods are more robust in autonomous driving [8,38]. Current 3D object detectors mainly represent the point cloud as raw points or voxels.…”

Section: Introductionmentioning

confidence: 99%

From Voxel to Point: IoU-guided 3D Object Detection for Point Cloud with Voxel-to-Point Decoder

Dai

Shao

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

In this paper, we present an Intersection-over-Union (IoU) guided two-stage 3D object detector with a voxel-to-point decoder. To preserve the necessary information from all raw points and maintain the high box recall in voxel based Region Proposal Network (RPN), we propose a residual voxel-to-point decoder to extract the point features in addition to the map-view features from the voxel based RPN. We use a 3D Region of Interest (RoI) alignment to crop and align the features with the proposal boxes for accurately perceiving the object position. The RoI-Aligned features are finally aggregated with the corner geometry embeddings that can provide the potentially missing corner information in the box refinement stage. We propose a simple and efficient method to align the estimated IoUs to the refined proposal boxes as a more relevant localization confidence. The comprehensive experiments on KITTI and Waymo Open Dataset demonstrate that our method achieves significant improvements with novel architectures against the existing methods. The code is available on Github URL 1 . CCS CONCEPTS• Computing methodologies → Object detection; Scene understanding; Vision for robotics.

show abstract

“…Ongoing development of various applications such as autonomous driving [5,11,32] and robotics [41] has led to the increasing demand for 3D object detectors that predict the class label, 3D bounding box and detection confidence score for each instance in a given 3D scene. In recent years, the 2D object detection methods have made great breakthroughs [6,15,22,49,55,59], far ahead of the 3D object detection methods, in particular, the anchor-free detection methods.…”

Section: Introductionmentioning

confidence: 99%

Anchor-free 3D Single Stage Detector with Mask-Guided Attention for Point Cloud

Dai

Shao

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

Most of the existing single-stage and two-stage 3D object detectors are anchor-based methods, while the efficient but challenging anchor-free single-stage 3D object detection is not well investigated. Recent studies on 2D object detection show that the anchor-free methods also are of great potential. However, the unordered and sparse properties of point clouds prevent us from directly leveraging the advanced 2D methods on 3D point clouds. We overcome this by converting the voxel-based sparse 3D feature volumes into the sparse 2D feature maps. We propose an attentive module to fit the sparse feature maps to dense mostly on the object regions through the deformable convolution tower and the supervised mask-guided attention. By directly regressing the 3D bounding box from the enhanced and dense feature maps, we construct a novel singlestage 3D detector for point clouds in an anchor-free manner. We propose an IoU-based detection confidence re-calibration scheme to improve the correlation between the detection confidence score and the accuracy of the bounding box regression. Our code is publicly available at https://github.com/jialeli1/MGAF-3DSSD. CCS CONCEPTS• Computing methodologies → Object detection; Scene understanding; Vision for robotics.

show abstract

Commands for Autonomous Vehicles by Progressively Stacking Visual-Linguistic Representations

Cited by 13 publications

References 24 publications

Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attention

Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attention

From Voxel to Point: IoU-guided 3D Object Detection for Point Cloud with Voxel-to-Point Decoder

Anchor-free 3D Single Stage Detector with Mask-Guided Attention for Point Cloud

Contact Info

Product

Resources

About