Selective Query-Guided Debiasing for Video Corpus Moment Retrieval

Yoon, Sunjae; Hong, Ji Woo; Eunseop, Yoon,; Dahyun, Kim,; Junyeong, Kim,; Yoon, Hyeun Joong; Yoo, Chang D.

doi:10.1007/978-3-031-20059-5_11

Cited by 5 publications

(3 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Following the popular metrics [23], [24] of video localization, we perform recall metrics about the intersection of union between the predicted temporal boundary and the ground-truth. Shortly, it is referred to as ''R@n,IoU=µ", where it denotes the percentage of targets having at least one prediction whose Intersection over Union (IoU) with ground truth is larger than µ in top-n localized temporal boundaries.…”

Section: B Experimental Details A: Evaluation Metricmentioning

confidence: 99%

Causal Localization Network for Radar Human Localization With Micro-Doppler Signature

Yoon,

Koo,

Shim

et al. 2024

IEEE Access

Self Cite

View full text Add to dashboard Cite

Micro-Doppler (MD) signature includes unique characteristics by different-sized body parts such as arms, legs, and torso. Existing radar identification systems have made an effort to classify the identification of humans using these characteristics presented in MD signatures while achieving a remarkable performance of classification. However, we argue that the radar identification system also should be extended to perform more fine-grained tasks to achieve the flexibility of the identification. In this paper, we introduce a radar human localization (RHL) task, which involves temporally localizing human identifications within untrimmed MD signatures. To enable RHL, we have constructed a micro-Doppler dataset referred to as IDRad-TBA. Furthermore, we propose Causal Localization Network (CLNet) as the RHL baseline system built upon the IDRad-TBA dataset. CLNet employs a novel temporal causal prediction approach for MD signature localization. Experimental results validate the effectiveness of CLNet in performing the RHL task. Our project is available at: https://github.com/dbstjswo505/CLNet INDEX TERMS deep learning, temporal human identification, micro-Doppler radar, information retrieval.

show abstract

Section: B Experimental Details A: Evaluation Metricmentioning

confidence: 99%

Causal Localization Network for Radar Human Localization With Micro-Doppler Signature

Yoon,

Koo,

Shim

et al. 2024

IEEE Access

Self Cite

View full text Add to dashboard Cite

show abstract

“…In this respect, we came up with two negative impacts: (1) unreliable vague information by conversational agents and (2) fairness issues in agents' responses. Therefore, word sense disambiguation techniques (Yoon et al, 2022a) and multimodal debiasing solutions (Yoon et al, 2022b;Niu et al, 2021) should also be applied to the dialogue systems.…”

Section: Ethics Statementmentioning

confidence: 99%

HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue

Yoon,

Kim,

Yoon

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Video-grounded Dialogue (VGD) aims to answer questions regarding a given multi-modal input comprising video, audio, and dialogue history. Although there have been numerous efforts in developing VGD systems to improve the quality of their responses, existing systems are competent only to incorporate the information in the video and text and tend to struggle in extracting the necessary information from the audio when generating appropriate responses to the question. The VGD system seems to be deaf, and thus, we coin this symptom of current systems' ignoring audio data as a deaf response. To overcome the deaf response problem, Hearing Enhanced Audio Response (HEAR) framework is proposed to perform sensible listening by selectively attending to audio whenever the question requires it. The HEAR framework enhances the accuracy and audibility of VGD systems in a model-agnostic manner. HEAR is validated on VGD datasets (i.e., AVSD@DSTC7 and AVSD@DSTC8) and shows effectiveness with various VGD systems.

show abstract

“…With the increasing abundance of video data, the ability to comprehend videos effectively has become exceedingly important. Thus far, notable progress has been made toward understanding videos that include video action detection/segmentation [1]- [3], video question answering [4], [5], video-grounded dialogue [6], [7], video moment retrieval [8] and video scene segmentation [9]- [15]. Among those, we focus on video scene segmentation (VSS), which plays a crucial role in understanding and interpreting long-term videos and can serve as the fundamental building block for AI systems designed to comprehend lengthy videos.…”

Section: Introductionmentioning

confidence: 99%

Contrasting Multi-Modal Similarity Framework for Video Scene Segmentation

Park,

Kim,

Seok

et al. 2024

IEEE Access

Self Cite

View full text Add to dashboard Cite

This paper proposes a video scene segmentation framework referred to as a Contrasting Multi-Modal Similarity (CMS). Video is composed of multiple scenes which are short stories or semantic units of video, with each scene consisting of multiple shots. The task of video scene segmentation aims to semantically segment long videos, such as movies, into the sequence of scenes by identifying the boundaries of each scene transition. Current video scene segmentation frameworks have primarily relied on comparing only the visual cues of adjacent shots to identify scene boundaries. These frameworks have focused on two major approaches: 1) comparing only the visual cues of adjacent frames to distinguish between scenes and 2) performing clustering based on visual cues for distinction among scenes. However, within videos, there exist numerous scenes that are difficult to distinguish using visual information alone, as they often appear similar or ambiguous. Taking inspiration from the aforementioned issues, we propose a framework referred to as CMS that leverages not only visual cues (i.e., shots) but also textual cues (i.e., captions) to semantically distinguish scenes. The new framework, CMS, leverages visual cues and text cues as follows: (1) Generate captions corresponding to each shot using a zero-shot captioning model (Caption Generation). ( 2) Construct similarity score matrices for each modality to measure semantic similarities (Similarity Score Calculation).(3) Based on the above matrix, select similar shots and dissimilar shots for contrastive training (Similarity Score-based Sampling). Our experiments show that the CMS framework advances the performance to exceed the previous state-of-the-art methods with a relatively simple approach without complex model architectures.

show abstract

Selective Query-Guided Debiasing for Video Corpus Moment Retrieval

Cited by 5 publications

References 23 publications

Causal Localization Network for Radar Human Localization With Micro-Doppler Signature

Causal Localization Network for Radar Human Localization With Micro-Doppler Signature

HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue

Contrasting Multi-Modal Similarity Framework for Video Scene Segmentation

Contact Info

Product

Resources

About