Visual Analysis for Wildlife Preserve based on Muti-systems

Lin, Lijing; Lü, Min; Li, Guozheng; Lai, Chufan; Jiang, Ruike; Liu, Qiangqiang; Yuan, Xiaoru

doi:10.1109/vast.2017.8585552

Cited by 1 publication

(2 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One line of research aims to augment a pre-trained language model with visual modality by learning a mapping from an external visual encoder to the frozen language model [2,20,26], demonstrating a zero-shot capability to perform multi-modal tasks leveraging knowledge stored in the language model. Recently, PaLI [6] fine-tuned on OK-VQA achieved SOTA performance and outperformed previous retrieval-based methods [11,27] by a substantial margin. However, our results on the INFOSEEK benchmark show that PaLI falls significantly behind pipeline systems with access to a knowledge base (KB), particularly on Time and Numerical questions.…”

Section: Related Workmentioning

confidence: 97%

“…Various approaches have been proposed to address knowledge-based VQA tasks [33] by incorporating external knowledge into visionlanguage models. One approach is to retrieve information from an external knowledge base [16,32,53] and employ fusion-in-decoder (FID) [17] to perform language QA [11,27]. Another approach is to transform the image into a text caption and use an LLM [5,7] to answer questions [15,56].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

Chen¹,

Hu²,

Luan³

et al. 2023

Preprint

View full text Add to dashboard Cite

Large language models [5,7] have demonstrated an emergent capability in answering knowledge intensive questions. With recent progress on web-scale visual and language pre-training [2,6,38], do these models also understand how to answer visual information seeking questions? To answer this question, we present INFOSEEK 1 , aVisual Question Answering dataset that focuses on asking information-seeking questions, where the information can not be answered by common sense knowledge. We perform a multi-stage human annotation to collect a natural distribution of high-quality visual information seeking questionanswer pairs. We also construct a large-scale, automatically collected dataset by combining existing visual entity recognition datasets and Wikidata, which provides over one million examples for model fine-tuning and validation.Based on INFOSEEK, we analyzed various pre-trained Visual QA systems to gain insights into the characteristics of different pre-trained models. Our analysis shows that it is challenging for the state-of-the-art multi-modal pre-trained models to answer visual information seeking questions, but this capability is improved through fine-tuning on the automated INFOSEEK dataset. We hope our analysis paves the way to understand and develop the next generation of multi-modal pre-training.

show abstract