SOMMA: Cortically inspired paradigms for multimodal processing

Lefort, Mathieu; Boniface, Yann; Girau, Bernard

doi:10.1109/ijcnn.2013.6706959

Cited by 7 publications

(11 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this article, we propose a paradigm for multiple data flows fusion by multimodal correlation learning. This problem was already addressed in other studies with various approaches as for example maximum covariance analysis by principal component analysis [14], combination of sensorymotor anticipations [15] or mutually constrained modal selforganizations [16] among others. The originality of our work lies mainly in its biological inspiration to provide a generic neural implementation with an emphasis on preferential mapping of predictable stimuli across modalities with separated modal processing.…”

Section: Introductionmentioning

confidence: 99%

PROPRE: PROjection and PREdiction for multimodal correlations learning. An application to pedestrians visual data discrimination

Lefort

Gepperth

2014

2014 International Joint Conference on Neural Networks (IJCNN)

Self Cite

View full text Add to dashboard Cite

PROPRE is a generic and modular unsupervised neural learning paradigm that extracts meaningful concepts of multimodal data flows based on predictability across modalities. It consists on the combination of three modules. First, a topological projection of each data flow on a self-organizing map. Second, a decentralized prediction of each projection activity from each others map activities. Third, a predictability measure that compares predicted and real activities. This measure is used to modulate the projection learning so that to favor the mapping of predictable stimuli across modalities. In this article, we use Kohonen map for the projection module, linear regression for the prediction one and we propose multiple generic predictability measures. We illustrate the properties and performances of PROPRE paradigm on a challenging supervised classification task of visual pedestrian data. The modulation of the projection learning by the predictability measure improves significantly classification performances of the system independently of the measure used. Moreover, PROPRE provides a combination of interesting functional properties, such as a dynamical adaptation to input statistic variations, that is rarely available in other machine learning algorithms.

show abstract

Section: Introductionmentioning

confidence: 99%

PROPRE: PROjection and PREdiction for multimodal correlations learning. An application to pedestrians visual data discrimination

Lefort

Gepperth

2014

2014 International Joint Conference on Neural Networks (IJCNN)

Self Cite

View full text Add to dashboard Cite

show abstract

“…It is also conceptually very close to the predictive coding model of hierarchical visual processing [17], [18], [19]. This focus on predictability and on generic multimodal processing are the two main points that distinguish our work from other multimodal self-organizing maps models [20], [21], [22].…”

Section: Introductionmentioning

confidence: 77%

Multimodal space representation driven by self-evaluation of predictability

Lefort

Kopinski

Gepperth

2014

4th International Conference on Development and Learning and on Epigenetic Robotics

Self Cite

View full text Add to dashboard Cite

PROPRE is a generic and modular neural learning paradigm that autonomously extracts meaningful concepts of multimodal data flows driven by predictability across modalities in an unsupervised, incremental and online way. For that purpose, PROPRE consists of the combination of projection and prediction. Firstly, each data flow is topologically projected with a self-organizing map, largely inspired from the Kohonen model. Secondly, each projection is predicted by each other map activities, by mean of linear regressions. The main originality of PROPRE is the use of a simple and generic predictability measure that compares predicted and real activities for each modal stream. This measure drives the corresponding projection learning to favor the mapping of predictable stimuli across modalities at the system level (i.e. that their predictability measure overcomes some threshold). This predictability measure acts as a self-evaluation module that tends to bias the representations extracted by the system so that to improve their correlations across modalities. We already showed that this modulation mechanism is able to bootstrap representation extraction from previously learned representations with artificial multimodal data related to basic robotic behaviors [1] and improves performance of the system for classification of visual data within a supervised learning context [2]. In this article, we improve the self-evaluation module of PROPRE, by introducing a sliding threshold, and apply it to the unsupervised classification of gestures caught from two timeof-flight (ToF) cameras. In this context, we illustrate that the modulation mechanism is still useful although less efficient than purely supervised learning.

show abstract

“…This consensus drives the self-organization in each module so that they perform a joint self-organization. This is the extension of previous work involving a cellular paradigm closer to the biological side [12,11] and leading to an abstraction of the neural field toward SOM-like structures, as initiated in [1]. This paper introduces self-organizing modules as well as a methodology for grouping them into an architecture.…”

Section: Introductionmentioning

confidence: 84%

Consensus Driven Self-Organization: Towards Non Hierarchical Multi-Map Architectures

Gonnier

Boniface

Frezza-Buet

2020

Communications in Computer and Information Science

Self Cite

View full text Add to dashboard Cite

This paper introduces CxSOM, a model to build modular architectures based on self-organizing maps (SOM). An original consensus driven approach enables to adress non-hierarchical architectures where SOMs get organized jointly. The paper aims at showing how the modules are able to store the association between data, and evaluating, by a mutual information criterion, the resulting organization. These results stand as preliminary work to study bigger architectures.

show abstract

SOMMA: Cortically inspired paradigms for multimodal processing

Cited by 7 publications

References 32 publications

PROPRE: PROjection and PREdiction for multimodal correlations learning. An application to pedestrians visual data discrimination

PROPRE: PROjection and PREdiction for multimodal correlations learning. An application to pedestrians visual data discrimination

Multimodal space representation driven by self-evaluation of predictability

Consensus Driven Self-Organization: Towards Non Hierarchical Multi-Map Architectures

Contact Info

Product

Resources

About