Towards artificial general intelligence via a multimodal foundation model

Fei, Nanyi; Lu, Zhiwu; Gao, Yizhao; Yang, Guo‐Yu; Huo, Yuqi; Wen, Jingyuan; Lu, Haoyu; Song, Ruihua; Gao, Xin; Xiang, Tao; Sun, Hao; Wen, Ji-Rong

doi:10.1038/s41467-022-30761-2

Cited by 106 publications

(40 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…45 Existing artificial neural network models also do not yet reach human performance on tasks involving multimodal integration. 39,40 For these reasons, future work to mimic the coding properties of anterior temporal lobe structures in artificial neural networks may enable machines to better approximate the remarkable human ability to learn concepts.…”

Section: Resultsmentioning

confidence: 99%

“…An important line of future investigation may be to explore whether the same experience-dependent changes occur in artificial neural networks that aim to learn multimodal object concepts. 39,40 Previous human neuroimaging has shown that the anterior temporal lobes are important for intra-object configural representations, 41 such that damage to the perirhinal cortex 14,42 leads to object discrimination impairment. For example, human participants with perirhinal cortex damage are unable to resolve feature-level interference created by viewing multiple objects with overlapping features.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal Object Representations Rely on Integrative Coding

Ladyka‐Wojcik

Qazilbash

et al. 2022

Preprint

View full text Add to dashboard Cite

Combining information from multiple senses is essential to object recognition. Yet how the mind combines sensory input into coherent multimodal representations, the multimodal binding problem, remains poorly understood. Here, we applied multi-echo fMRI across a four-day paradigm, in which participants learned 3-dimensional multimodal object representations created from well-characterized visual shape and sound features. Our novel paradigm decoupled the learned multimodal object representations from their baseline unimodal shape and sound features, thus tracking the emergence of multimodal concepts as they were learned by healthy adults. Critically, the representation for the whole object was different from the combined representation of its individual parts, with evidence of an integrative object code in anterior temporal lobe structures. Intriguingly, the perirhinal cortex, an anterior temporal lobe structure, was by default biased towards visual shape, but this initial shape bias was attenuated with learning. Pattern similarity analyses suggest that after learning the perirhinal cortex orthogonalized combinations of visual shape and sound features, transforming overlapping feature input into distinct multimodal object representations. These results provide evidence of integrative coding in the anterior temporal lobes that is distinct from the distributed sensory features, advancing the age-old question of how the mind constructs multimodal objects from their component features.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Multimodal Object Representations Rely on Integrative Coding

Ladyka‐Wojcik

Qazilbash

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…6). Recently, (Fei et al, 2022) have employed a similar technique to visualize what they call the "imagination" of a model trained to bring closer an image and a matching textual description in a higher-level feature space. In a similar way, my imaging consciousness could try to strongly re-activate the processes that allow me to recognize (in perception) my friend as being "blond, tall, with a snub or aquiline nose, etc.".…”

Section: Sartrean Imagination: Conscious Re-presentation Of Sedimente...mentioning

confidence: 99%

Rejecting Cognitivism: Computational Phenomenology for Deep Learning

Beckmann¹,

Köstner²,

Hipólito³

2023

Preprint

View full text Add to dashboard Cite

We propose a non-representationalist framework for deep learning relying on a novel method: computational phenomenology, a dialogue between the first-person perspective (relying on phenomenology) and the mechanisms of computational models. We thereby reject the modern cognitivist interpretation of deep learning, according to which artificial neural networks encode representations of external entities. This interpretation mainly relies on neuro-representationalism, a position that combines a strong ontological commitment towards scientific theoretical entities and the idea that the brain operates on symbolic representations of these entities. We proceed as follows: after offering a review of cognitivism and neuro-representationalism in the field of deep learning, we first elaborate a phenomenological critique of these positions; we then sketch out computational phenomenology and distinguish it from existing alternatives; finally we apply this new method to deep learning models trained on specific tasks, in order to formulate a conceptual framework of deep-learning, that allows one to think of artificial neural networks' mechanisms in terms of lived experience.

show abstract

“…In [33], to enhance the performance of transferring models in a zero-shot fashion, LiT, namely Lockedimage Tuning, was proposed using contrastive-tuning methods. BriVL (Bridging-Vision-and-Language) was proposed in [34] as a method for obtaining multi-cognitive abilities and developing general AI by leveraging weak semantic correlation data. More details about big AI models can be found in [23].…”

Section: B Big Ai Modelsmentioning

confidence: 99%

Parallel Factories for Smart Industrial Operations: From Big AI Models to Field Foundational Models and Scenarios Engineering

Wang

Cheng

et al. 2022

IEEE/CAA J. Autom. Sinica

View full text Add to dashboard Cite

The rapid advancement of fundamental theories and computing capacity has brought artificial intelligence, internet of things, extended reality, and many other new intelligent technologies into our daily lives. Due to the lack of interpretability and reliability guarantees, it is extremely challenging to apply these technologies directly to real-world industrial systems. Here we present a new paradigm for establishing parallel factories in metaverses to accelerate the deployment of intelligent technologies in real-world industrial systems: QAII-1.0. Based on cyber-physical-social systems, QAII-1.0 incorporates complex social and human factors into the design and analysis of industrial operations and is capable of handling industrial operations involving complex social and human behaviors. In QAII-1.0, a field foundational model called EuArtisan combined with scenarios engineering is developed to improve the intelligence of industrial systems while ensuring industrial interpretability and reliability. Finally, parallel oil fields in metaverses are established to demonstrate the operating procedure of QAII-1.0.

show abstract

Towards artificial general intelligence via a multimodal foundation model

Cited by 106 publications

References 34 publications

Multimodal Object Representations Rely on Integrative Coding

Multimodal Object Representations Rely on Integrative Coding

Rejecting Cognitivism: Computational Phenomenology for Deep Learning

Parallel Factories for Smart Industrial Operations: From Big AI Models to Field Foundational Models and Scenarios Engineering

Contact Info

Product

Resources

About