Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar

Scheiner, Nicolas; Kraus, Florian; Wei, Fangyin; Phan, Buu; Mannan, Fahim; Appenrodt, Nils; Ritter, Walter; Dickmann, Jürgen; Dietmayer, Klaus; Sick, Bernhard; Heide, Felix

doi:10.1109/cvpr42600.2020.00214

Cited by 105 publications

(60 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Increasingly, deep learning is used for audio classification [27], [28], and localization [29] of sources in line-of-sight, in which case visual detectors can replace manual labeling [30], [31]. Analogous to our work, [32] presents a first deep learning method for sensing around corners but with automotive radar. Thus, while the effect of occlusions on sensor measurements is difficult to model [14], data-driven approaches appear to be a good alternative.…”

Section: Related Workmentioning

confidence: 85%

Hearing What You Cannot See: Acoustic Vehicle Detection Around Corners

Schulz¹,

Mattar²,

Hehn³

et al. 2021

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

This work proposes to use passive acoustic perception as an additional sensing modality for intelligent vehicles. We demonstrate that approaching vehicles behind blind corners can be detected by sound before such vehicles enter in line-ofsight. We have equipped a research vehicle with a roof-mounted microphone array, and show on data collected with this sensor setup that wall reflections provide information on the presence and direction of occluded approaching vehicles. A novel method is presented to classify if and from what direction a vehicle is approaching before it is visible, using as input Direction-of-Arrival features that can be efficiently computed from the streaming microphone array data. Since the local geometry around the ego-vehicle affects the perceived patterns, we systematically study several environment types, and investigate generalization across these environments. With a static ego-vehicle, an accuracy of 0.92 is achieved on the hidden vehicle classification task. Compared to a state-of-the-art visual detector, Faster R-CNN, our pipeline achieves the same accuracy more than one second ahead, providing crucial reaction time for the situations we study. While the ego-vehicle is driving, we demonstrate positive results on acoustic detection, still achieving an accuracy of 0.84 within one environment type. We further study failure cases across environments to identify future research directions.

show abstract

Section: Related Workmentioning

confidence: 85%

Hearing What You Cannot See: Acoustic Vehicle Detection Around Corners

Schulz¹,

Mattar²,

Hehn³

et al. 2021

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

show abstract

“…Often, it could be sufficient to be able to detect objects and track their motion. Thanks to a greatly reduced number of degrees of freedom, this problem can be addressed with less detailed input data and even steady-state (intensity, no time of flight) input images under passive [Bouman et al 2017] or active [Chen et al 2019;] illumination, and it has led to the first industry demonstrators to integrate robust non-line-of-sight sensing technology [Scheiner et al 2020]. Nonetheless, with an expensive bill of components, these demonstrators are unlikely to converge to mass-market products.…”

Section: Related Workmentioning

confidence: 99%

Low-cost SPAD sensing for non-line-of-sight tracking, material classification and depth imaging

et al. 2021

Self Cite

View full text Add to dashboard Cite

Fig. 1. We propose the use of cheap, small off-the-shelf distance sensors (far left) for a variety of computational imaging and vision tasks, and demonstrate and evaluate their capabilities in emerging sensing applications like (from left to right) material classification, non-line-of-sight tracking, and depth imaging.Time-correlated imaging is an emerging sensing modality that has been shown to enable promising application scenarios, including lidar ranging, fluorescence lifetime imaging, and even non-line-of-sight sensing. A leading technology for obtaining time-correlated light measurements are singlephoton avalanche diodes (SPADs), which are extremely sensitive and capable of temporal resolution on the order of tens of picoseconds. However, the rare and expensive optical setups used by researchers have so far prohibited these novel sensing techniques from entering the mass market. Fortunately, SPADs also exist in a radically cheaper and more power-efficient version that has been widely deployed as proximity sensors in mobile devices for almost a decade. These commodity SPAD sensors can be obtained at a mere few cents per detector pixel. However, their inferior data quality and severe technical drawbacks compared to their high-end counterparts necessitate the use of additional optics and suitable processing algorithms. In this paper, we adopt an existing evaluation platform for commodity SPAD sensors, and modify it to unlock time-of-flight (ToF) histogramming and hence computational imaging. Based on this platform, we develop and demonstrate a family of hardware/software systems that, for the first time, implement applications that had so far been limited to significantly more advanced, higher-priced setups: direct ToF depth imaging, non-line-of-sight object tracking, and material classification.

show abstract

“…NLOS imaging have used a variety of sensing technologies at the convergence of physics, optics, electronics, and signal processing. Optical detectors, including optical interferometry [24], PMDs [19], SPADs [4,5,20], intensified CCD cameras (iCCDs) [25], streak cameras [3,9,13,18,26], or even non-optical devices, e.g., acoustic [27], thermal [28], and radar [29], are exploited to collect transients of hidden objects. The temporal resolution of a detector is one of the key parameters for NLOS capture systems.…”

Section: Related Workmentioning

confidence: 99%

“…In Equation (24), we notice that the temporal jitter 𝑗 (𝑡; 𝜎, 𝛾) dominates the measurement effect with a SPAD when decomposing the Poisson distribution from 𝜏 SPAD . The Wiener filter can thus be applied to denoise the transients 𝜏 SPAD : τ(𝜈; s) = k (𝜈) τSPAD (𝜈; s) (29) where τ(𝜈; s) and τSPAD represent the Fourier transform of 𝜏(𝑡; s) and 𝜏 SPAD (𝑡; s), while 𝜈 is the frequency. The kernel of the Wiener filter k (𝜈) in the frequency domain is computed from the Fourier transform of the temporal jitter, j (𝜈), and the signal-to-noise ratio 𝜂, as:…”

Section: Transient Enhancementmentioning

confidence: 99%

Onsite Non-Line-of-Sight Imaging via Online Calibrations

Pan¹,

Li²,

Gao³

et al. 2021

Preprint

View full text Add to dashboard Cite

There has been an increasing interest in deploying non-line-of-sight (NLOS) imaging systems for recovering objects behind an obstacle. Existing solutions generally pre-calibrate the system before scanning the hidden objects. Onsite adjustments of the occluder, object and scanning pattern require re-calibration. We present an online calibration technique that directly decouples the acquired transients at onsite scanning into the LOS and hidden components. We use the former to directly (re)calibrate the system upon changes of scene/obstacle configurations, scanning regions, and scanning patterns whereas the latter for hidden object recovery via spatial, frequency or learning based techniques. Our technique avoids using auxiliary calibration apparatus such as mirrors or checkerboards and supports both laboratory validations and realworld deployments.

show abstract

Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar

Cited by 105 publications

References 47 publications

Hearing What You Cannot See: Acoustic Vehicle Detection Around Corners

Hearing What You Cannot See: Acoustic Vehicle Detection Around Corners

Low-cost SPAD sensing for non-line-of-sight tracking, material classification and depth imaging

Onsite Non-Line-of-Sight Imaging via Online Calibrations

Contact Info

Product

Resources

About