Finding Fallen Objects Via Asynchronous Audio-Visual Integration

Gan, Chuang; Gu, Yi; Zhou, Siyuan; Schwartz, J.; Alter, Seth; Traer, James; Gutfreund, Dan; Tenenbaum, Joshua B.; McDermott, Josh H.; Torralba, Antonio

doi:10.1109/cvpr52688.2022.01027

Cited by 8 publications

(1 citation statement)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The direct application is a sound synthesis model adapted to nonlinear musical instruments with low computational cost and can yield realtime event-driven synthesis of sounds in virtual or augmented reality environments, a particularly active field of research [28] [29]. For example, Gan proposes an interactive multimodal simulation platform based on impact sounds [30] [31] that could be compatible with the model proposed in our study. The outcomes of such a study can be useful for sonic interaction design [32] and the synthesis of new sounds [33].…”

Section: Introductionmentioning

confidence: 99%

A Perceptually Evaluated Signal Model: Collisions Between a Vibrating Object and an Obstacle

Poirot

Bilbao

Aramaki

et al. 2023

IEEE/ACM Trans. Audio Speech Lang. Process.

View full text Add to dashboard Cite

The collision interaction mechanism between a vibrating string and a non-resonant obstacle is at the heart of many musical instruments. This paper focuses on the identification of perceptually salient auditory features related to this phenomenon. The objective is to design a signal-based synthesis process, with an eye towards developing intuitive control strategies. To this end, a database of synthesized sounds is assembled through physics-based emulation of a string/obstacle collision, in order to characterize the effect of collisions on time-frequency content. The investigation of this database reveals characteristic timefrequency patterns related to the position of the obstacle during the interaction. In particular, a frequency shift of certain modes is apparent for strong interactions, which, alongside the generation of new frequency components, leads to increased perceived roughness and inharmonicity. These observations enable the design of a real-time compatible signal-based sound synthesis process, with a mapping of synthesis parameters linked to the perceived location of the obstacle. The accuracy of the signal model with respect to the physical model sound output and recorded sounds was evaluated through listening tests: time-frequency patterns reproduced by the signal model enabled listeners to precisely recognize the transverse location of the obstacle.

show abstract

Section: Introductionmentioning

confidence: 99%