Image captioning for effective use of language models in knowledge-based visual question answering

Salaberria, Ander; Azkune, Gorka; Lacalle, Oier López de; Soroa, Aitor; Agirre, Eneko

doi:10.1016/j.eswa.2022.118669

Cited by 33 publications

(9 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The advantage of this approach is that it can generate specific answers to specific questions. However, the limitations of this approach are that it requires large amounts of labeled data and it may not capture the context and complexity of the video content [38], [39], [40]. Video Retrieval using LLMs refers to the process of searching and retrieving relevant videos from a large video database using advanced language models.…”

Section: Methodsmentioning

confidence: 99%

A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming

Zhou,

Wang,

Liu

et al. 2024

Preprint

View full text Add to dashboard Cite

This paper offers an insightful examination of how currently top-trending AI technologies, i.e., generative artificial intelligence (Generative AI) and large language models (LLMs), are reshaping the field of video technology, including video generation, understanding, and streaming. It highlights the innovative use of these technologies in producing highly realistic videos, a significant leap in bridging the gap between real-world dynamics and digital creation. The study also delves into the advanced capabilities of LLMs in video understanding, demonstrating their effectiveness in extracting meaningful information from visual content, thereby enhancing our interaction with videos. In the realm of video streaming, the paper discusses how LLMs contribute to more efficient and user-centric streaming experiences, adapting content delivery to individual viewer preferences. This comprehensive review navigates through the current achievements, ongoing challenges, and future possibilities of applying Generative AI and LLMs to video-related tasks, underscoring the immense potential these technologies hold for advancing the field of video technology related to multimedia, networking, and AI communities.Impact Statement-This paper contributes to the field of video technology by examining the integration of Generative AI and Large Language Models (LLMs) in video generation, understanding, and streaming. Its exploration of these technologies offers a foundational understanding of their potential and limitations in enhancing the realism and interactivity of video content. The exploration of LLMs in video comprehension sets the stage for advancements in accessibility and interaction, promising enhanced educational tools, improved user interfaces, and advanced video analytics applications. Additionally, the paper underscores the role of LLMs in optimizing video streaming services, leading to more personalized and bandwidth-efficient platforms. This could substantially benefit the entertainment sector with adaptive streaming solutions tailored to individual preferences. By identifying key challenges and future research directions, the paper guides ongoing efforts to merge AI with video technology, while raising awareness about potential ethical issues. Its influence extends beyond academia, encouraging responsible AI development and policy-making in video technology, balancing technological advancements with ethical considerations.

show abstract

Section: Methodsmentioning

confidence: 99%

A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming

Zhou,

Wang,

Liu

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Attention mechanism mainly focuses on selective actions/things related to tasks and ignores other irrelevant actions/things. Researchers are working on designing an effective attention-based neural network for vision-related applications such as fine-grained image recognition [107,108], image classification [109,110], image captioning [111,112], and vehicle re-identification [113]. The process of vehicle re-identification based on spatiotemporal attention is shown in Figure 9.…”

Section: Vehicle Re-identification Based On Attention Mechanismmentioning

confidence: 99%

Multi-Object Multi-Camera Tracking Based on Deep Learning for Intelligent Transportation: A Review

Fei

Han

2023

Sensors

View full text Add to dashboard Cite

Multi-Objective Multi-Camera Tracking (MOMCT) is aimed at locating and identifying multiple objects from video captured by multiple cameras. With the advancement of technology in recent years, it has received a lot of attention from researchers in applications such as intelligent transportation, public safety and self-driving driving technology. As a result, a large number of excellent research results have emerged in the field of MOMCT. To facilitate the rapid development of intelligent transportation, researchers need to keep abreast of the latest research and current challenges in related field. Therefore, this paper provide a comprehensive review of multi-object multi-camera tracking based on deep learning for intelligent transportation. Specifically, we first introduce the main object detectors for MOMCT in detail. Secondly, we give an in-depth analysis of deep learning based MOMCT and evaluate advanced methods through visualisation. Thirdly, we summarize the popular benchmark data sets and metrics to provide quantitative and comprehensive comparisons. Finally, we point out the challenges faced by MOMCT in intelligent transportation and present practical suggestions for the future direction.

show abstract

“…Nevertheless, the context-free nature of classic word embedding methods [69,70] can only serve a limited amount of cases, impeding generalization to scenarios when contextualization is necessary. Transformers leveraged on the linguistic side allowed further improvements on K-VQA models [71,72] paving the way for consequent end-to-end VL approaches.…”

Section: Visual Question Answering (Vqa)mentioning

confidence: 99%

The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges

Lymperaiou¹,

Stamou²

2023

Preprint

View full text Add to dashboard Cite

Recent advancements in visiolinguistic (VL) learning have allowed the development of multiple models and techniques that offer several impressive implementations, able to currently resolve a variety of tasks that require the collaboration of vision and language. Current datasets used for VL pre-training only contain a limited amount of visual and linguistic knowledge, thus significantly limiting the generalization capabilities of many VL models. External knowledge sources such as knowledge graphs (KGs) and Large Language Models (LLMs) are able to cover such generalization gaps by filling in missing knowledge, resulting in the emergence of hybrid architectures. In the current survey, we analyze tasks that have benefited from such hybrid approaches. Moreover, we categorize existing knowledge sources and types, proceeding to discussion regarding the KG vs LLM dilemma and its potential impact to future hybrid approaches.

show abstract

Image captioning for effective use of language models in knowledge-based visual question answering

Cited by 33 publications

References 6 publications

A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming

A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming

Multi-Object Multi-Camera Tracking Based on Deep Learning for Intelligent Transportation: A Review

The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges

Contact Info

Product

Resources

About