A dual‐channel language decoding from brain activity with progressive transfer training

Abstract

When we view a scene, the visual cortex extracts and processes visual information in the scene through various kinds of neural activities. Previous studies have decoded the neural activity into single/multiple semantic category tags which can caption the scene to some extent. However, these tags are isolated words with no grammatical structure, insufficiently conveying what the scene contains. It is well-known that textual language (sentences/phrases) is superior to single word in disclosing the meaning of images as well as reflecting people's real understanding of the images. Here, based on artificial intelligence technologies, we attempted to build a dual-channel language decoding model (DC-LDM) to decode the neural activities evoked by images into language (phrases or short sentences). The DC-LDM consisted of five modules, namely, Image-Extractor, Image-Encoder, Nerve-Extractor, Nerve-Encoder, and Language-Decoder. In addition, we employed a strategy of progressive transfer to train the DC-LDM for improving the performance of language decoding. The results showed that the texts decoded by DC-LDM could describe natural image stimuli accurately and vividly. We adopted six indexes to quantitatively evaluate the difference between the decoded texts and the annotated texts of corresponding visual images, and found that Word2vec-Cosine similarity (WCS) was the best indicator to reflect the similarity between the decoded and the annotated texts. In addition, among different visual cortices, we found that the text decoded by the higher visual cortex was more consistent with the description of the natural image than the lower one. Our decoding model may provide enlightenment in language-based brain-computer interface explorations.

From Sight to Insight: A Multi-task Approach with the Visual Language Decoding Model

“…Third, our multitask visual-language decoding model succeeded in accurately capturing key objects and actions within images, organizing them into fluent and comprehensible textual descriptions. This echoes well with our prior work which directly utilized a dual-channel language model to generate precise language information from fMRI neural activity 20 . Furthermore, our findings indirectly corroborated that higher brain areas, such as FFA and VWFA, are primarily responsible for handling high-level semantic information, while lower-level brain regions like V1 and V2 handle fundamental visual information 41–43 .…”

“…Further, to investigate preferences within the visual cortex for language decoding, we assessed ten distinct regions (including V1, V2, V3, EBA, FBA, FFA, OFA, OPA, PPA, and VWFA) along with three combined areas (LVC, HVC, and VC) for their decoding accuracy using Word2vec. We chose Word2vec due to its demonstrated effectiveness in assessing language relevance and measuring semantic similarity 20 .…”

“…Since 2005, renowned labs worldwide have made significant progress in visual decoding using machine learning and deep learning techniques. These techniques are capable of extracting multi-level linguistic information including object categories 14,15 , semantic labels 16,17 and descriptive text 18–20 from visual activities triggered by natural scenes. However, these researches mainly decode information independently through a single task, which lacks knowledge sharing between different decoding tasks as well as limits the generalization capability of visual decoding.…”

Emotional Content and Semantic Structure of Dialogues Are Associated With Interpersonal Neural Synchrony in the Prefrontal Cortex

“…To do so, we introduce a quantitative framework investigating the two dimensions of dialogues’ emotional content and syntactic/semantic structure for predicting interpersonal neu-ral synchrony during naturalistic social exchanges. Our study, thus, bridges innovative cognitive neuroscience data [40] with affective computing and cognitive data science frameworks [30], integrating mind and brain data [41, 42]. In particular, we lever-age the recent modeling framework of Textual Forma Mentis Networks (TFMNs) for representing conceptual associations encoded in text and retrieved via artificial intelli-gence (AI) and psychologically validated data [32, 43].…”

Boosting-GNN: Boosting Algorithm for Graph Networks on Imbalanced Node Classification

“…Convolutional neural networks (CNNs) have been widely used in image recognition (Russakovsky et al, 2015 ; He et al, 2016 ), object detection (Lin et al, 2014 ), speech recognition (Yu et al, 2016 ), visual coding and decoding (Huang et al, 2021a , b ). However, traditional CNNs can only handle data in the Euclidean space.…”