End-to-End Video Question-Answer Generation With Generator-Pretester Network

Su, Hung-Ting; Chang, Chen-Hsi; Shen, Po-Wei; Wang, Yu-Siang; Chang, Ya-Liang; Chang, Yu‐Cheng; Cheng, Po-Hsun; Hsu, Winston H.

doi:10.1109/tcsvt.2021.3051277

Cited by 30 publications

(6 citation statements)

References 59 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Video QG is an upcoming area, where some works have been introduced lately. In [ 128 ], using a newly constructed architecture of a generator, which generates a question given a video clip and an answer, and a pre-tester, which tries to answer the generated question, has been investigated for joint QA-QG from videos. They make use of Video encoder for extracting video features of 20 frames from each video and later encode them using faster R-CNN and Resnet101.…”

Section: Challenges and Future Directionsmentioning

confidence: 99%

Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications

Mulla¹,

Gharpure

2023

Prog Artif Intell

View full text Add to dashboard Cite

Question generation in natural language has a wide variety of applications. It can be a helpful tool for chatbots for generating interesting questions as also for automating the process of question generation from a piece of text. Most modern-day systems, which are conversational, require question generation ability for identifying the user’s needs and serving customers better. Generating questions in natural language is now, a more evolved task, which also includes generating questions for an image or video. In this review, we provide an overview of the research progress in automatic question generation. We also present a comprehensive literature review covering the classification of Question Generation systems by categorizing them into three broad use-cases, namely standalone question generation, visual question generation, and conversational question generation. We next discuss the datasets available for the same for each use-case. We further direct this review towards applications of question generation and discuss the challenges in this field of research.

show abstract

Section: Challenges and Future Directionsmentioning

confidence: 99%

Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications

Mulla¹,

Gharpure

2023

Prog Artif Intell

View full text Add to dashboard Cite

show abstract

“…Some works explore this task in text using techniques such as pipeline (Subramanian et al, 2018), multi-agent system (Wang et al, 2019), hierarchical variational model (Subramanian et al, 2018) or coreference knowledge (Lee et al, 2020). Su et al (2021) also proposes a model for QAP generation from video. However, such QAP generation works assume answers are selected from the spans of input context (Subramanian et al, 2018;Lee et al, 2020;Wang et al, 2019) or the given candidates (Su et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

“…Su et al (2021) also proposes a model for QAP generation from video. However, such QAP generation works assume answers are selected from the spans of input context (Subramanian et al, 2018;Lee et al, 2020;Wang et al, 2019) or the given candidates (Su et al, 2021). As answers could not be extracted directly from images and there are no candidate ones, the above methods can not be simply applied to the image.…”

Section: Related Workmentioning

confidence: 99%

Diversity and Consistency: Exploring Visual Question-Answer Pair Generation

Yang

Zhou

Feng

et al. 2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

Although showing promising values to downstream applications, generating question and answer together is under-explored. In this paper, we introduce a novel task that targets question-answer pair generation from visual images. It requires not only generating diverse question-answer pairs but also keeping the consistency of them. We study different generation paradigms for this task and propose three models: the pipeline model, the joint model, and the sequential model. We integrate variational inference into these models to achieve diversity and consistency. We also propose region representation scaling and attention alignment to improve the consistency further. We finally devise an evaluator as a quantitative metric for consistency. We validate our approach on two benchmarks, VQA2.0 and Visual-7w, by automatically and manually evaluating diversity and consistency. Experimental results show the effectiveness of our models: they can generate diverse or consistent pairs. Moreover, this task can be used to improve visual question generation and visual question answering.

show abstract

“…answer [1], [2], [3], [4]. Further, video captioning uses the relational modals to generate captions for related videos, it has recently shifted to dense video captioning and produces many captions simultaneously as compared to one caption [5].…”

Section: Introductionmentioning

confidence: 99%

Problematic Unordered Queries in Temporal Moment Measurement by Using Natural Language

Nawaz

Dong

2023

IEEE Access

View full text Add to dashboard Cite

This study examines the difficulty in measuring temporal moments by using natural language (TMMNL) in the untrimmed video. The purpose of TMMNL is to use natural language query to find a specific moment within a lengthy video. It's a challenging task since other, closely related activities may divert attention from the target temporal moment. This issue has been addressed by existing research employing computer vision techniques like reinforcement, anchor, and ranking. In this research, we not only propose a TMMNL solution and show how to use natural language query to find the required moment, but we also identify a novel issue: if the given natural language query is unordered (without proper subject, verb and object), the system will have trouble understanding and the network may perform poorly. Previous methods perform poorer when a query is unordered and cannot able to retrieve the relevant moment as a result overall performance degradation. We introduce novel the-visual, the-action, the-object, and theconnecting words concept to address the problem of unordered query in TMMNL. Graph Convolutions with Latent variable for Visual-Textual Network (GCL-VTN) is our suggested network, which has three components: (1) visual-graph-convolution (visual GC), (2) textual-graph-convolution (textual GC), and (3) compatible method for learning embedding's (CMLE). Visual-nodes in the visual GC detect regional attributes, object, and actor information in the same way as textual-nodes in the textual GC maintain word sequence using grammar-based query rules. A compatible method for learning embedding (CMLE) is also proposed, which integrates different modalities (moment, query) and trained grammar-based words into the same embedding space. In order to align and keep the query sequence, we also incorporate a stochastic latent variable in the CMLE that has prior and posterior distributions. The posterior distribution deals with both visual-textual data and works when the query is in the correct sequence or based on grammar rules. The prior distribution only deals with textual data and is effective when the query is unordered or not based on grammar rules. TACoS, Charades-STA, and activityNet-Captions are state-of-the-art; our GCL-VTN exceeds them all.INDEX TERMS Moment measurement via language query in video, temporal moment measurement , Moment measurement , Cross-Modal Interactions, Single Moment Retrieval.

show abstract

End-to-End Video Question-Answer Generation With Generator-Pretester Network

Cited by 30 publications

References 59 publications

Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications

Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications

Diversity and Consistency: Exploring Visual Question-Answer Pair Generation

Problematic Unordered Queries in Temporal Moment Measurement by Using Natural Language

Contact Info

Product

Resources

About