Commands 4 Autonomous Vehicles (C4AV) Workshop Summary

Deruyttere, Thierry; Vandenhende, Simon; Grujicic, Dusan; Liu, Yu; Gool, Luc Van; Blaschko, Matthew B.; Tuytelaars, Tinne; Moens, Marie‐Francine

doi:10.1007/978-3-030-66096-3_1

Cited by 6 publications

(8 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We project the top-32 confidence scoring predictions to the frontal view and evaluate the rate at which at least one sample achieves an IoU ≥ 0.5 with the ground truth referred object in the Talk2Car test set. We find that such a bounding box exists in 92% of the cases which is the same as with the bounding boxes of (Deruyttere et al 2020), and therefore, no substantial increase in IoU is expected solely from improved bounding box predictions. The average distance between the 3D bounding box of the ground truth referred object and their 3D predicted bounding is 1.7m on the Talk2Car test set.…”

Section: D Object Detectorsupporting

confidence: 56%

Predicting Physical World Destinations for Commands Given to Self-Driving Cars

Grujicic

Deruyttere

Moens

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

In recent years, we have seen significant steps taken in the development of self-driving cars. Multiple companies are starting to roll out impressive systems that work in a variety of settings. These systems can sometimes give the impression that full self-driving is just around the corner and that we would soon build cars without even a steering wheel. The increase in the level of autonomy and control given to an AI provides an opportunity for new modes of human-vehicle interaction. However, surveys have shown that giving more control to an AI in self-driving cars is accompanied by a degree of uneasiness by passengers. In an attempt to alleviate this issue, recent works have taken a natural language-oriented approach by allowing the passenger to give commands that refer to specific objects in the visual scene. Nevertheless, this is only half the task as the car should also understand the physical destination of the command, which is what we focus on in this paper. We propose an extension in which we annotate the 3D destination that the car needs to reach after executing the given command and evaluate multiple different baselines on predicting this destination location. Additionally, we introduce a model that outperforms the prior works adapted for this particular setting.

show abstract

Section: D Object Detectorsupporting

confidence: 56%

Predicting Physical World Destinations for Commands Given to Self-Driving Cars

Grujicic

Deruyttere

Moens

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

“…An extra experiment where we heavily undersample the training set further supports this point and reveals that ten samples per category suffice to attain performances close to those using the full data set. Overall, the low computational cost and high predictive power of principled canonical representations support their potential use as a compact scene descriptor for applications that require fast real-time scene understanding, such as realized by self-driving cars [12].…”

Section: Volume X 2021mentioning

confidence: 98%

“…Only the image is provided, and no 32 information about the Object classes is given. One must 12 A few images in HICO-DET have more than one O class -the majority of images have a unique object class. There are often multiple instances of the same object class (e.g., bikes) and / or multiple triplet classes t with the same O involved (e.g., (human, repairing, bike) and (human, holding, bike)).…”

Section: Layout Transformationsmentioning

confidence: 99%

“…16 Overall, the low computational burden and effectiveness shown by spatial clues pave the way toward using them as low-dimensional scene descriptors to perform fast scene understanding at a glance, in a human-like scene "gist" fashion. Concretely, applications that require fast real-time scene understanding, such as automatic object localization from natural language commands in self-driving cars [12] show particular promise. Furthermore, the strong performance of our methods with small training data sets showcases the potential for problems with a small amount of data.…”

Section: Experiment: Reducing Data Sizementioning

confidence: 99%

See 1 more Smart Citation

Probing Spatial Clues: Canonical Spatial Templates for Object Relationship Understanding

2021

Self Cite

View full text Add to dashboard Cite

Humans often leverage spatial clues to categorize scenes in a fraction of a second. This form of intelligence is very relevant in time-critical situations (e.g., when driving a car) and valuable to transfer to automated systems. This work investigates the predictive power of solely processing spatial clues for scene understanding in 2D images and compares such an approach with the predictive power of visual appearance. To this end, we design the laboratory task of predicting the identity of two objects (e.g., "man" and "horse") and their relationship or predicate (e.g., "riding") given exclusively the ground truth bounding box coordinates of both objects. We also measure the performance attainable in Human Object Interaction (HOI) detection, a real-world spatial task, which includes a setting where ground truth boxes are not available at test time. An additional goal is to identify the principles necessary to effectively represent a spatial template, that is, the visual region in which two objects involved in a relationship expressed by a predicate occur. We propose a scale-, mirror-, and translation-invariant representation that captures the spatial essence of the relationship, i.e., a canonical spatial representation. Tests in two benchmarks reveal:(1) High performance is attainable by using exclusively spatial information in all tasks. (2) In HOI detection, the canonical template outperforms the rest of spatial, visual, and several state-of-the-art baselines. (3) Simple fusion of visual and spatial features substantially improves performance. (4) Our methods fare remarkably well with a small amount of data and rare categories. Our results obtained on the Visual Genome (VG) and the Humans Interacting with Common Objects -Detection (HICO-DET) datasets indicate that great predictive power can be obtained from spatial clues alone, opening up possibilities for performing fast scene understanding at a glance.INDEX TERMS Spatial understanding, spatial layout, computer vision, vision and scene understanding. I. INTRODUCTION 1A Well-researched concept in cognitive science is the 2 gist, or the initial representation of a scene obtained 3 in a brief glance. The gist may include semantic content 4 (e.g., "is a classroom"), the identity of a few objects (e.g., 5"there are books"), and the spatial layout [1]. Humans can 6 categorize scenes in a fraction of a second (∼13-250 ms) 7[1], [2]. Generally, more detailed scenes and finer-grained 8

show abstract

“…When passengers give a command to a self-driving car that refers to a specific object, a first step in the command understanding is to detect this object (Deruyttere et al, 2020b). This task is often called visual grounding (VG) in the literature.…”

Section: Introductionmentioning

confidence: 99%

Giving Commands to a Self-Driving Car: How to Deal with Uncertain Situations?

Deruyttere,

Milewski,

Moens

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Current technology for autonomous cars primarily focuses on getting the passenger from point A to B. Nevertheless, it has been shown that passengers are afraid of taking a ride in self-driving cars. One way to alleviate this problem is by allowing the passenger to give natural language commands to the car. However, the car can misunderstand the issued command or the visual surroundings which could lead to uncertain situations. It is desirable that the self-driving car detects these situations and interacts with the passenger to solve them. This paper proposes a model that detects the uncertain situations when a command is given and finds the visual objects causing it. Optionally, a question generated by the system describing the uncertain objects is included. We argue that if the car could explain the objects in a human-like way, passengers could gain more confidence in the car's abilities. Thus, we investigate how to (1) detect uncertain situations and their underlying causes, and (2) how to generate clarifying questions for the passenger. When evaluating on the Talk2Car dataset, we show that the proposed model, Uncertainty Resolving System (URS), improves 12.6% in terms of IoU .5 compared to not using URS. Furthermore, we designed a referring expression generator (REG) Attribute-Referring Expression Generator (A-REG) tailored to a self-driving car setting which yields a relative improvement of 6% METEOR and 8% ROUGE-l compared with state-of-the-art REG models, and is three times faster.

show abstract

Commands 4 Autonomous Vehicles (C4AV) Workshop Summary

Cited by 6 publications

References 41 publications

Predicting Physical World Destinations for Commands Given to Self-Driving Cars

Predicting Physical World Destinations for Commands Given to Self-Driving Cars

Probing Spatial Clues: Canonical Spatial Templates for Object Relationship Understanding

Giving Commands to a Self-Driving Car: How to Deal with Uncertain Situations?

Contact Info

Product

Resources

About