Multimodal Chain-of-Thought Reasoning in Language Models

Zhang, Zhuosheng; Zhang, Aston; Zhang, Hai; Karypis, George; Smola, Alex

doi:10.48550/arxiv.2302.00923

Cited by 34 publications

(50 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, LLaMA-Adapter can be easily switched into multi-modal variant and achieves +10% higher accuracy. Besides, we notice that MM-CoT [59] is on par with our approach, but it relies on the complex two-stage inference. We believe our LLaMA-Adapter can also be boosted and leave the exploration of chain-of-thought for future research.…”

Section: Performancementioning

confidence: 98%

See 1 more Smart Citation

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

Zhang¹,

Han²,

Zhou³

et al. 2023

Preprint

View full text Add to dashboard Cite

We present LLaMA-Adapter, a lightweight adaption method to efficiently fine-tune LLaMA into an instructionfollowing model. Using 52K self-instruct demonstrations, LLaMA-Adapter only introduces 1.2M learnable parameters upon the frozen LLaMA 7B model, and costs less than one hour for fine-tuning on 8 A100 GPUs. Specifically, we adopt a set of learnable adaption prompts, and prepend them to the input text tokens at higher transformer layers. Then, a zero-init attention mechanism with zero gating is proposed, which adaptively injects the new instructional cues into LLaMA, while effectively preserves its pre-trained knowledge. With efficient training, LLaMA-Adapter generates high-quality responses, comparable to Alpaca with fully fine-tuned 7B parameters. Furthermore, our approach can be simply extended to multi-modal input, e.g., images, for image-conditioned LLaMA, which achieves superior reasoning capacity on ScienceQA. We release our code at https://github.com/ZrrSkywalker/LLaMA-Adapter.

show abstract

Section: Performancementioning

confidence: 98%

“…In Table 2, we compare LLaMA-Adapter with popular VQA methods [2,9,17,18,22,23,28,53] and language models [5,16,59]. As shown, our single-modal variant ('LLaMA-Adapter T ') attains 78.31% accuracy with 1.2M parameters.…”

Section: Performancementioning

confidence: 99%

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

Zhang¹,

Han²,

Zhou³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Reasoning is a crucial component of human intelligence that enables us to draw inferences, make decisions, and solve complex problems. However, even trained with large scale dataset, sometimes GAI models could still fail at common sense reasoning tasks [256,257]. Recently, more and more researchers began to focus on this problem.…”

Section: Privacymentioning

confidence: 99%

“…By incorporating this approach, large language models can achieve higher accuracy and better performance in tasks that require logical reasoning. CoT has also been applied to other areas like vision language question answering [257] and code generation [258]. However, it still remains a problem that how to construct these CoT prompts according to specific tasks.…”

Section: Privacymentioning

confidence: 99%

A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT

Cao¹,

Li²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recently, ChatGPT, along with DALL-E-2 [1] and Codex [2],has been gaining significant attention from society. As a result, many individuals have become interested in related resources and are seeking to uncover the background and secrets behind its impressive performance. In fact, ChatGPT and other Generative AI (GAI) techniques belong to the category of Artificial Intelligence Generated Content (AIGC), which involves the creation of digital content, such as images, music, and natural language, through AI models. The goal of AIGC is to make the content creation process more efficient and accessible, allowing for the production of high-quality content at a faster pace. AIGC is achieved by extracting and understanding intent information from instructions provided by human, and generating the content according to its knowledge and the intent information. In recent years, large-scale models have become increasingly important in AIGC as they provide better intent extraction and thus, improved generation results. With the growth of data and the size of the models, the distribution that the model can learn becomes more comprehensive and closer to reality, leading to more realistic and high-quality content generation. This survey provides a comprehensive review on the history of generative models, and basic components, recent advances in AIGC from unimodal interaction and multimodal interaction. From the perspective of unimodality, we introduce the generation tasks and relative models of text and image. From the perspective of multimodality, we introduce the cross-application between the modalities mentioned above. Finally, we discuss the existing open problems and future challenges in AIGC.

show abstract

“…Specifically, [101] tackles VCR by captioning the image, and then feed the caption together with the existing linguistic input to the LLM. Another promising work in this direction introduces Multimodal-CoT without using language as the mediating modality, proposing a two-stage process to separately infer the answer A and the rationale R, while stating that a LM with less than 1B parameters is adequate for state-of-the-art performance [102]. It is expected that the rapid rise of popularity of LLMs in complex linguistic QA reasoning [103] may soon give rise to more LLM-augmented VCR approaches, addressing more aspects of reasoning.…”

Section: Visual Commonsense Reasoning (Vcr)mentioning

confidence: 99%

The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges

Lymperaiou¹,

Stamou²

2023

Preprint

View full text Add to dashboard Cite

Recent advancements in visiolinguistic (VL) learning have allowed the development of multiple models and techniques that offer several impressive implementations, able to currently resolve a variety of tasks that require the collaboration of vision and language. Current datasets used for VL pre-training only contain a limited amount of visual and linguistic knowledge, thus significantly limiting the generalization capabilities of many VL models. External knowledge sources such as knowledge graphs (KGs) and Large Language Models (LLMs) are able to cover such generalization gaps by filling in missing knowledge, resulting in the emergence of hybrid architectures. In the current survey, we analyze tasks that have benefited from such hybrid approaches. Moreover, we categorize existing knowledge sources and types, proceeding to discussion regarding the KG vs LLM dilemma and its potential impact to future hybrid approaches.

show abstract

Multimodal Chain-of-Thought Reasoning in Language Models

Cited by 34 publications

References 16 publications

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention

A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT

The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges

Contact Info

Product

Resources

About