Online joint learning of object concepts and language model using multimodal hierarchical Dirichlet process

Aoki, Tatsuya; Nishihara, Joe; Nagaoka, Tomoaki; Nagai, Takayuki

doi:10.1109/iros.2016.7759410

Cited by 9 publications

(11 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The point of this model is that multimodal categories are used for learning lexical information and vice versa. This way, the robot could acquire approximately 70 appropriate words through interaction with a user over 1 month [86]. This suggests that unsupervised learning, e.g., multimodal object categorization, can provide a basic internal representation system for primal language learning.…”

Section: Unsupervised Learning Viewpoint: From Multimodal Categorimentioning

confidence: 99%

Symbol Emergence in Cognitive Developmental Systems: A Survey

Taniguchi

Piater

Wörgötter

et al. 2019

IEEE Trans. Cogn. Dev. Syst.

Self Cite

View full text Add to dashboard Cite

Humans use signs, e.g., sentences in a spoken language, for communication and thought. Hence, symbol systems like language are crucial for our communication with other agents and adaptation to our real-world environment. The symbol systems we use in our human society adaptively and dynamically change over time. In the context of artificial intelligence (AI) and cognitive systems, the symbol grounding problem has been regarded as one of the central problems related to symbols. However, the symbol grounding problem was originally posed to connect symbolic AI and sensorimotor information and did not consider many interdisciplinary phenomena in human communication and dynamic symbol systems in our society, which semiotics considered. In this paper, we focus on the symbol emergence problem, addressing not only cognitive dynamics but also the dynamics of symbol systems in society, rather than the symbol grounding problem. We first introduce the notion of a symbol in semiotics from the humanities, to leave the very narrow idea of symbols in symbolic AI. Furthermore, over the years, it became more and more clear that symbol emergence has to be regarded as a multifaceted problem. Therefore, secondly, we review the history of the symbol emergence problem in different fields, including both biological and artificial systems, showing their mutual relations. We summarize the discussion and provide an integrative viewpoint and comprehensive overview of symbol emergence in cognitive systems. Additionally, we describe the challenges facing the creation of cognitive systems that can be part of symbol emergence systems. Fig. 1. Robot in a home environment that has to deal with complex manipulation, planning, and interaction via semiotic communication with human users.

show abstract

Section: Unsupervised Learning Viewpoint: From Multimodal Categorimentioning

confidence: 99%

Symbol Emergence in Cognitive Developmental Systems: A Survey

Taniguchi

Piater

Wörgötter

et al. 2019

IEEE Trans. Cogn. Dev. Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the experiments, we used the dataset that was used in (Aoki et al, 2016). This dataset was composed of images, tactile sensor values, and sound data obtained by the robot from objects and utterances given by a human who teaches features of the objects.…”

Section: Datasetmentioning

confidence: 99%

“…Furthermore, the integrated model can generate images from the features that are estimated from the words using the MLDA. We used a multimodal dataset composed of images and teaching utterances, which were obtained by the robot observing the objects and the human teacher teaching the object features using speech (Aoki et al, 2016). Because the human teacher did not necessarily utter the words corresponding to the object labels, the teaching utterances included words that were not related to the labels, and speech recognition errors consequently occurred.…”

Section: Introductionmentioning

confidence: 99%

Robot Concept Acquisition Based on Interaction Between Probabilistic and Deep Generative Models

Kuniyasu¹,

Nagaoka²,

Taniguchi³

et al. 2021

Front. Comput. Sci.

Self Cite

View full text Add to dashboard Cite

We propose a method for multimodal concept formation. In this method, unsupervised multimodal clustering and cross-modal inference, as well as unsupervised representation learning, can be performed by integrating the multimodal latent Dirichlet allocation (MLDA)-based concept formation and variational autoencoder (VAE)-based feature extraction. Multimodal clustering, representation learning, and cross-modal inference are critical for robots to form multimodal concepts from sensory data. Various models have been proposed for concept formation. However, in previous studies, features were extracted using manually designed or pre-trained feature extractors and representation learning was not performed simultaneously. Moreover, the generative probabilities of the features extracted from the sensory data could be predicted, but the sensory data could not be predicted in the cross-modal inference. Therefore, a method that can perform clustering, feature learning, and cross-modal inference among multimodal sensory data is required for concept formation. To realize such a method, we extend the VAE to the multinomial VAE (MNVAE), the latent variables of which follow a multinomial distribution, and construct a model that integrates the MNVAE and MLDA. In the experiments, the multimodal information of the images and words acquired by a robot was classified using the integrated model. The results demonstrated that the integrated model can classify the multimodal information as accurately as the previous model despite the feature extractor learning in an unsupervised manner, suitable image features for clustering can be learned, and cross-modal inference from the words to images is possible.

show abstract

“…In particular, Araki et al (2012b) proposed online Multimodal Latent Dirichlet Allocation (oMLDA) to acquire object concepts in an online manner, and combined this with the Nested Pitman-Yor Language Model (NPYLM), making it possible to perform lexical acquisition of unknown words sequentially. Aoki et al (2016) constructed an algorithm that can infer an approximately global optimal solution by representing it as a single integrated model. The NPYLM is an unsu-pervised morphological analysis method based on a statistical model that enables word segmentation exclusively from phoneme sequences (Mochihashi et al 2009).…”

Section: Improvement Of Online Learning Based On Particle Filters In mentioning

confidence: 99%

Improved and scalable online learning of spatial concepts and language models with mapping

et al. 2020

View full text Add to dashboard Cite

We propose a novel online learning algorithm, called SpCoSLAM 2.0, for spatial concepts and lexical acquisition with high accuracy and scalability. Previously, we proposed SpCoSLAM as an online learning algorithm based on unsupervised Bayesian probabilistic model that integrates multimodal place categorization, lexical acquisition, and SLAM. However, our original algorithm had limited estimation accuracy owing to the influence of the early stages of learning, and increased computational complexity with added training data. Therefore, we introduce techniques such as fixed-lag rejuvenation to reduce the calculation time while maintaining an accuracy higher than that of the original algorithm. The results show that, in terms of estimation accuracy, the proposed algorithm exceeds the original algorithm and is comparable to batch learning. In addition, the calculation time of the proposed algorithm does not depend on the amount of training data and becomes constant for each step of the scalable algorithm. Our approach will contribute to the realization of long-term spatial language interactions between humans and robots.

show abstract

Online joint learning of object concepts and language model using multimodal hierarchical Dirichlet process

Cited by 9 publications

References 20 publications

Symbol Emergence in Cognitive Developmental Systems: A Survey

Symbol Emergence in Cognitive Developmental Systems: A Survey

Robot Concept Acquisition Based on Interaction Between Probabilistic and Deep Generative Models

Improved and scalable online learning of spatial concepts and language models with mapping

Contact Info

Product

Resources

About