In this paper, we introduce a framework that merges classical ideas borrowed from scale-space and multiresolution segmentation with nonlinear partial differential equations. A non-linear scale-space stack is constructed by means of an appropriate diffusion equation. This stack is analyzed and a tree of coherent segments is constructed based on relationships between different scale layers. Pruning this tree proves to be a very efficient tool for unsupervised segmentation of different classes of images (e.g., natural, medical, etc.). This technique is light on the computational point of view and can be extended to nonscalar data in a straightforward manner.

This paper presents a novel method to correlate audio and visual data generated by the same physical phenomenon, based on sparse geometric representation of video sequences. The video signal is modeled as a sum of geometric primitives evolving through time, that jointly describe the geometric and motion content of the scene. The displacement through time of relevant visual features, like the mouth of a speaker, can thus be compared with the evolution of an audio feature to assess the correspondence between acoustic and visual signals. Experiments show that the proposed approach allows to detect and track the speaker's mouth when several persons are present on the scene, in presence of distracting motion, and without prior face or mouth detection.

No abstract

No abstract

In this work we explore the potentialities of a framework for the representation of audio-visual signals using decompositions on overcomplete dictionaries. Redundant decompositions may describe audio-visual sequences in a concise fashion, preserving good representation properties thanks to the use of redundant, well designed, dictionaries. We expect that this will help us overcome two typical problems of multimodal fusion algorithms. On one hand, classical representation techniques, like pixel-based measures (for the video) or Fourier-like transforms (for the audio), take into account only marginally the physics of the problem. On the other hand, the input signals have large dimensionality. The results we obtain by making use of sparse decompositions of audio-visual signals over redundant codebooks are encouraging and show the potentialities of the proposed approach to multimodal signal representation.