Camera Pose Estimation and Localization with Active Audio Sensing

Yang, Karren; Firman, Michael; Brachmann, Eric; Godard, Clément

doi:10.1007/978-3-031-19836-6_16

Cited by 8 publications

(6 citation statements)

References 95 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multi-modal learning has enabled a variety of applications for embodied agents, i.e., autonomously moving agents. For example, there are audio-visual navigation, depth estimation, and camera pose estimation [4]- [12]. In particular, several methods have been proposed to process multiple sound sources by taking advantage of the agent's movement, even though the sound input is binaural [7]- [9].…”

Section: C: Multi-modal Learning For Embodied Agentsmentioning

confidence: 99%

“…On the other hand, focusing on the fact that most devices are equipped with multiple sensors such as cameras and microphones, various models utilizing multi-modal input have been proposed [4]- [12]. These methods include those that leverage the movement of the devices themselves to analyze scenes with multiple sound sources, even with devices like binaural microphones that have insufficient spatial information.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Direction-of-Arrival Estimation for Mobile Agents Utilizing the Relationship Between Agent’s Trajectory and Binaural Audio

Sato,

Sugano,

Sato

2024

IEEE Access

View full text Add to dashboard Cite

With the development of robotics and wearable devices, there is a need for information processing under the assumption that an agent itself is mobile. Especially, understanding an acoustic environment around an agent is an important issue. In this paper, we solve a task in which a moving agent estimates the Direction of Arrival (DoA) of the surrounding sound sources. To this end, we propose a novel training method, Trajectory-based Direction Selection (TDS). In TDS, a mixture of binaural audio recorded by two agents and their trajectories are given as input to a network. Then, the network is trained to estimate the DoA of surrounding sounds that correspond to each agent's trajectory separately. By corresponding the agent's trajectory to the binaural audio with TDS, we can estimate DoAs of multiple sounds even with binaural audio as audio input, which has not been realized by sound-only methods. In simulated environments covering both single and multiple sources, our method outperforms existing DoA estimation methods.INDEX TERMS Audio processing, Direction of Arrival estimation, embodied agents, multi-modal learning

show abstract

Section: C: Multi-modal Learning For Embodied Agentsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Direction-of-Arrival Estimation for Mobile Agents Utilizing the Relationship Between Agent’s Trajectory and Binaural Audio

Sato,

Sugano,

Sato

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…In contrast, we learn camera pose and sound localization solely from self-supervision, obtaining angular predictions without labeled data. Other work uses echolocation sounds to learn representations [27,93] and predict depth maps [19,68] and estimate camera poses [93] using labeled data. In contrast, our proposed approach jointly learns binaural sound localization and camera pose through passive audio sensing, without supervision.…”

Section: Related Workmentioning

confidence: 99%

Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation

Chen¹,

Qian²,

Owens³

2023

Preprint

View full text Add to dashboard Cite

The images and sounds that we perceive undergo subtle but geometrically consistent changes as we rotate our heads. In this paper, we use these cues to solve a problem we call Sound Localization from Motion (SLfM): jointly estimating camera rotation and localizing sound sources. We learn to solve these tasks solely through self-supervision. A visual model predicts camera rotation from a pair of images, while an audio model predicts the direction of sound sources from binaural sounds. We train these models to generate predictions that agree with one another. At test time, the models can be deployed independently. To obtain a feature representation that is well-suited to solving this challenging problem, we also propose a method for learning an audio-visual representation through cross-view binauralization: estimating binaural sound from one view, given images and sound from another. Our model can successfully estimate accurate rotations on both real and synthetic scenes, and localize sound sources with accuracy competitive with state-of-the-art self-supervised approaches. Project site: https://ificl.github.io/SLfM .

show abstract

“…We evaluate our method on Replica [79], 12-Scenes [31], and 7-Scenes [20] datasets. Replica contains high-fidelity indoor scenes and is widely used by recent works of NeRFs and localization [80]- [83]. We use the sequences recorded in [80], choosing the first sequence of each scene for training and the second for testing.…”

Section: B Camera Pose Estimationmentioning

confidence: 99%

Hallucinated Neural Radiance Fields in the Wild

Chen

Zhang

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Neural Radiance Fields (NeRF) have been proposed for photorealistic novel view rendering. However, it requires many different views of one scene for training. Moreover, it has poor generalizations to new scenes and requires retraining or finetuning on each scene. In this paper, we develop a new NeRF model for novel view synthesis using only a single image as input. We propose to combine the (coarse) planar rendering and the (fine) volume rendering to achieve higher rendering quality and better generalizations. We also design a depth teacher net that predicts dense pseudo depth maps to supervise the joint rendering mechanism and boost the learning of consistent 3D geometry. We evaluate our method on three challenging datasets. It outperforms state-of-the-art single-view NeRFs by achieving 5∼20% improvements in PSNR and reducing 20∼50% of the errors in the depth rendering. It also shows excellent generalization abilities to unseen data without the need to finetune on each new scene.

show abstract

Camera Pose Estimation and Localization with Active Audio Sensing

Cited by 8 publications

References 95 publications

Direction-of-Arrival Estimation for Mobile Agents Utilizing the Relationship Between Agent’s Trajectory and Binaural Audio

Direction-of-Arrival Estimation for Mobile Agents Utilizing the Relationship Between Agent’s Trajectory and Binaural Audio

Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation

Hallucinated Neural Radiance Fields in the Wild

Contact Info

Product

Resources

About