Unmasked Teacher: Towards Training-Efficient Video Foundation Models

Li, Kunchang; Wang, Yali; Li, Yizhuo; Wang, Yi; He, Yuanqing; Wang, Limin; Qiao, Yu

doi:10.48550/arxiv.2303.16058

Cited by 2 publications

(2 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…LLMs are able to provide semantic information and generate symbolic spatial signals, which can serve as guidance for video scene understanding. Recently this has been demonstrated for interactive video-based dialogue and conversation [42], [41], [83], [90], [46], [91]. In this context, Video-ChatGPT [42] is designed for video understanding and conversation by capturing the spatial-temporal relationships between video frames based on LLMs.…”

Section: B Video Scene Understandingmentioning

confidence: 99%

A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming

Zhou,

Wang,

Liu

et al. 2024

Preprint

View full text Add to dashboard Cite

This paper offers an insightful examination of how currently top-trending AI technologies, i.e., generative artificial intelligence (Generative AI) and large language models (LLMs), are reshaping the field of video technology, including video generation, understanding, and streaming. It highlights the innovative use of these technologies in producing highly realistic videos, a significant leap in bridging the gap between real-world dynamics and digital creation. The study also delves into the advanced capabilities of LLMs in video understanding, demonstrating their effectiveness in extracting meaningful information from visual content, thereby enhancing our interaction with videos. In the realm of video streaming, the paper discusses how LLMs contribute to more efficient and user-centric streaming experiences, adapting content delivery to individual viewer preferences. This comprehensive review navigates through the current achievements, ongoing challenges, and future possibilities of applying Generative AI and LLMs to video-related tasks, underscoring the immense potential these technologies hold for advancing the field of video technology related to multimedia, networking, and AI communities.Impact Statement-This paper contributes to the field of video technology by examining the integration of Generative AI and Large Language Models (LLMs) in video generation, understanding, and streaming. Its exploration of these technologies offers a foundational understanding of their potential and limitations in enhancing the realism and interactivity of video content. The exploration of LLMs in video comprehension sets the stage for advancements in accessibility and interaction, promising enhanced educational tools, improved user interfaces, and advanced video analytics applications. Additionally, the paper underscores the role of LLMs in optimizing video streaming services, leading to more personalized and bandwidth-efficient platforms. This could substantially benefit the entertainment sector with adaptive streaming solutions tailored to individual preferences. By identifying key challenges and future research directions, the paper guides ongoing efforts to merge AI with video technology, while raising awareness about potential ethical issues. Its influence extends beyond academia, encouraging responsible AI development and policy-making in video technology, balancing technological advancements with ethical considerations.

show abstract

Section: B Video Scene Understandingmentioning

confidence: 99%

A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming

Zhou,

Wang,

Liu

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…For this purpose, several video foundation models (InternVideo [6], mPLUG [7], UnMasked Teacher [8]) were analyzed within such machine learning tasks, which can be divided into two groups:…”

Section: Main Part Artificial Intelligence Based Proctoring Systems A...mentioning

confidence: 99%

Human action analysis models in artificial intelligence based proctoring systems and dataset for them

Antoshchuk¹,

Breskina²

2023

AAIT

View full text Add to dashboard Cite

This paper describes the approach for building a specialized model for human action analysis in AI-based proctoring systemsand proposes a prototype of dataset which contains data specific to the application area. Boosted development of machine learningtechnologies, the availability of devicesand the access to the Internet are skyrocketing the development of the field of distance learning. And in parallel with distance learning systemstheAI-based proctoring systems, that provide the functional analysis of student work by imitating the teacher's assessment, are developing as well. However, despite the development of image processing and machine learning technology, the functionality of modern proctoring systems is still at a primitive level. Within the image processing functionality, they focus entirely on tracking students' faces and do not track postures and actions. At the same time, assessment of physical activity is necessary not only as part of the learning process, but also to keep students healthy according to regulatory requirements, as they spend the entire duration of learning process in front of computers or other devices during the distance learning. In existing implementations, this process falls entirely on the shoulders of teachers or even the studentsthemselves, who work through the lesson materials or tests on their own. Teachers, at the same time, have toeither establish contact through video communication systems and social media (TikTok, Instagram) and/or analyse videos of students doing certain physical activities in ordertoorganise physical activities evaluation. The lack of such functionality in AI-based proctoring systemsslows down the learning process and potentially harms students' health in the long run. This paper presents additional functionality requirements forAI-based proctoring systemsincluding human action analysis functionality to assess physical activity and to monitor hygiene rules for working with computers during the educational process. For this purpose, afoundation modelcalledInternVideo was used for processing and analysis of student's actions. Based on it, the approach for building a specialized model for student action analysis was proposed. It includes two modes of student activity evaluation during the distance learning process: static and dynamic. The static mode (aka working phase) analyses and evaluates the student's behavior during the learning and examination process, where physical activity is not the main component of learning. The dynamic mode (aka physical education mode) analyses and assesses the student who purposefully performs physical activity (physical education lesson, exercises for children during the lesson, etc.). A prototype dataset designed specifically for this application area has also been proposed.

show abstract

Unmasked Teacher: Towards Training-Efficient Video Foundation Models

Cited by 2 publications

References 48 publications

A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming

A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming

Human action analysis models in artificial intelligence based proctoring systems and dataset for them

Contact Info

Product

Resources

About