No abstract

Active speaker detection is an important component in video analysis algorithms for applications such as speaker diarization, video re-targeting for meetings, speech enhancement, and human-robot interaction. The absence of a large, carefully labeled audio-visual dataset for this task has constrained algorithm evaluations with respect to data diversity, environments, and accuracy. This has made comparisons and improvements difficult. In this paper, we present the AVA Active Speaker detection dataset (AVA-ActiveSpeaker) that will be released publicly to facilitate algorithm development and enable comparisons. The dataset contains temporally labeled face tracks in video, where each face instance is labeled as speaking or not, and whether the speech is audible. This dataset contains about 3.65 million human labeled frames or about 38.5 hours of face tracks, and the corresponding audio. We also present a new audio-visual approach for active speaker detection, and analyze its performance, demonstrating both its strength and the contributions of the dataset.

Speech activity detection (or endpointing) is an important processing step for applications such as speech recognition, language identification and speaker diarization. Both audio-and vision-based approaches have been used for this task in various settings, often tailored toward end applications. However, much of the prior work reports results in synthetic settings, on taskspecific datasets, or on datasets that are not openly available. This makes it difficult to compare approaches and understand their strengths and weaknesses. In this paper, we describe a new dataset which we will release publicly containing densely labeled speech activity in YouTube videos 1 , with the goal of creating a shared, available dataset for this task. The labels in the dataset annotate three different speech activity conditions: clean speech, speech co-occurring with music, and speech cooccurring with noise, which enable analysis of model performance in more challenging conditions based on the presence of overlapping noise. We report benchmark performance numbers on AVA-Speech using off-the-shelf, state-of-the-art audio and vision models that serve as a baseline to facilitate future research.

This study starts from the counterintuitive question of how we can render conventional stiff, nonstretchable, and even brittle materials sufficiently conformable to fully wrap curved surfaces, such as spheres, without failure. Here, we extend the geometrical design method of computational origami to wrapping. Our computational wrapping approach provides a robust and reliable method for fabricating conformal devices for arbitrary curved surfaces with a computationally designed nonpolyhedral developable net. This computer-aided design transforms two-dimensional (2D)–based materials, such as Si wafers and steel sheets, into various targeted conformal structures that can fully wrap desired 3D structures without fracture or severe plastic deformation. We further demonstrate that our computational wrapping approach enables a design platform that can transform conventional nonstretchable 2D-based devices, such as electroluminescent lighting and flexible batteries, into conformal 3D curved devices.

No abstract