Deconstructing multimodality: visual properties and visual context in human semantic processing

Davis, Charles C.; Bulat, Luana; Vero, Anita Lilla; Shutova, Ekaterina

doi:10.18653/v1/s19-1013

Cited by 6 publications

(5 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Based on the concepts and experimental results of psychology and neuroscience, there have been attempts to solve the problem of neural networks. In particular, the theory of mind, inductive bias, and intrinsic motivation were effective methods in embodied visual language interaction [190][191][192].…”

Section: ) Multi-agent Optimizationmentioning

confidence: 99%

A Metaverse: Taxonomy, Components, Applications, and Open Challenges

2022

View full text Add to dashboard Cite

Unlike previous studies on the Metaverse based on Second Life, the current Metaverse is based on the social value of Generation Z that online and offline selves are not different. With the technological development of deep learning-based high-precision recognition models and natural generation models, Metaverse is being strengthened with various factors, from mobile-based always-on access to connectivity with reality using virtual currency. The integration of enhanced social activities and neural-net methods requires a new definition of Metaverse suitable for the present, different from the previous Metaverse. This paper divides the concepts and essential techniques necessary for realizing the Metaverse into three components (i.e., hardware, software, and contents) and three approaches (i.e., user interaction, implementation, and application) rather than marketing or hardware approach to conduct a comprehensive analysis. Furthermore, we describe essential methods based on three components and techniques to Metaverse's representative Ready Player One, Roblox, and Facebook research in the domain of films, games, and studies. Finally, we summarize the limitations and directions for implementing the immersive Metaverse as social influences, constraints, and open challenges.

show abstract

Section: ) Multi-agent Optimizationmentioning

confidence: 99%

A Metaverse: Taxonomy, Components, Applications, and Open Challenges

2022

View full text Add to dashboard Cite

show abstract

“…More recently, some work has explored the quality of representations learned from images only (Lüddecke et al, 2019) or by combining language, vision, and emojis (Rotaru and Vigliocco, 2020). In parallel, new evaluation methods based, for example, on decoding brain activity (Davis et al, 2019) or success on tasks such as image retrieval (Kottur et al, 2016) have been proposed. This mass of studies has overall demonstrated the effectiveness of multimodal representations in approximating human semantic intuitions better than purely textual ones.…”

Section: Evaluating Multimodal Representationsmentioning

confidence: 99%

Word Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation

Pezzelle

Takmaz

Fernández

2021

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

This study carries out a systematic intrinsic evaluation of the semantic representations learned by state-of-the-art pre-trained multimodal Transformers. These representations are claimed to be task-agnostic and shown to help on many downstream language-and-vision tasks. However, the extent to which they align with human semantic intuitions remains unclear. We experiment with various models and obtain static word representations from the contextualized ones they learn. We then evaluate them against the semantic judgments provided by human speakers. In line with previous evidence, we observe a generalized advantage of multimodal representations over language- only ones on concrete word pairs, but not on abstract ones. On the one hand, this confirms the effectiveness of these models to align language and vision, which results in better semantic representations for concepts that are grounded in images. On the other hand, models are shown to follow different representation learning patterns, which sheds some light on how and when they perform multimodal integration.

show abstract

“…This dataset includes at least one label for all but four of the 21,841 synsets used in ImageNet. ImageNet has been used to solve tasks in the intersection of language and vision research Chen et al, 2019;Davis et al, 2019;Vempala and Preot, 2019). For Arabic computer vision, limited related work currently exists.…”

Section: ‫كلب‬mentioning

confidence: 99%

Proceedings of the First Workshop on Advances in Language and Vision Research

2020

View full text Add to dashboard Cite

Workshop schedule details: https://alvr-workshop.github.ioThe workshop also holds the first Video-guided Machine Translation (VMT) challenge and the REVERIE challenge. The VMT challenge aims to benchmark progress towards models that translate source language sentence into the target language with video information as the additional spatiotemporal context. The challenge is based on the recently released large-scale multilingual video description dataset, VA-TEX. The VATEX dataset contains over 41,250 videos and 825,000 high-quality captions in both English and Chinese, half of which are English-Chinese translation pairs. The REVERIE challenge requires an intelligent agent to correctly localize a remote target object (cannot be observed at the starting location) specified by a concise high-level natural language instruction, such as "bring me the blue cushion from the sofa in the living room". Since the target object is in a different room from the starting one, the agent needs first to navigate to the goal location. When the agent determines to stop, it should select one object from a list of candidates provided by the simulator. The agent can attempt to localize the target at any step, which is totally up to algorithm design. But the agent is only allowed to output once in each episode, which means the agent only can guess the answer once in a single run.

show abstract

Deconstructing multimodality: visual properties and visual context in human semantic processing

Cited by 6 publications

References 23 publications

A Metaverse: Taxonomy, Components, Applications, and Open Challenges

A Metaverse: Taxonomy, Components, Applications, and Open Challenges

Word Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation

Proceedings of the First Workshop on Advances in Language and Vision Research

Contact Info

Product

Resources

About