Dense-Captioning Events in Videos

Krishna, Ranjay; Hata, Kenji; Ren, Frederic; Li, Feifei; Niebles, Juan Carlos

doi:10.1109/iccv.2017.83

Cited by 1,052 publications

(1,016 citation statements)

References 66 publications

Supporting

Mentioning

1,009

Contrasting

Unclassified

Order By: Relevance

“…The objective is to extract a set of informative frames in order to briefly summarize the video content. (3) The video caption datasets are annotated with descried sentences or phrases, which can be based on either a trimmed video [75], [77] or different segments of a long video [35]. Our COIN is relevant to the above mentioned datasets, as it requires to localize the temporal boundaries of important steps corresponding to a task.…”

Section: Datasets Related To Instructional Video Analysismentioning

confidence: 99%

See 1 more Smart Citation

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Tang

Lu²,

Zhou³

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Thanks to the substantial and explosively inscreased instructional videos on the Internet, novices are able to acquire knowledge for completing various tasks. Over the past decade, growing efforts have been devoted to investigating the problem on instructional video analysis. However, the most existing datasets in this area have limitations in diversity and scale, which makes them far from many real-world applications where more diverse activities occur. To address this, we present a large-scale dataset named as "COIN" for COmprehensive INstructional video analysis. Organized with a hierarchical structure, the COIN dataset contains 11,827 videos of 180 tasks in 12 domains (e.g., vehicles, gadgets, etc.) related to our daily life. With a new developed toolbox, all the videos are annotated efficiently with a series of step labels and the corresponding temporal boundaries. In order to provide a benchmark for instructional video analysis, we evaluate plenty of approaches on the COIN dataset under five different settings. Furthermore, we exploit two important characteristics (i.e., task-consistency and ordering-dependency) for localizing important steps in instructional videos. Accordingly, we propose two simple yet effective methods, which can be easily plugged into conventional proposal-based action detection models. We believe the introduction of the COIN dataset will promote the future in-depth research on instructional video analysis for the community. Our dataset, annotation toolbox and source code are available at http://coin-dataset.github.io. DomainTaskStep VehiclesHousehold Items Change the Car Tire {unscrew the screws, jack up the car, remove the tire, put on the tire, tighten the screws } {remove the door knob, remove bolt and pin board, install new pin board, install new bolt, install new door knob } Replace the Door Knob

show abstract

Section: Datasets Related To Instructional Video Analysismentioning

confidence: 99%

“…The toolbox has two modes: frame mode and video mode. The frame mode is new developed for efficient annotation, while the video mode is frequently used in previous works [35]. We have evaluated the annotation time on a small set of COIN, which contains 25 videos of 7 tasks.…”

Section: Appendix B Annotation Time Cost Analysismentioning

confidence: 99%

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Tang

Lu²,

Zhou³

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…video retrieval by language, a large number of datasets have been proposed [36,1,26,19,31,30,29,33]. ActivityNet Captions [19] is a dataset with dense captions describing videos from ActivityNet [3], which can facilitate tasks such as video retrieval and temporal localization with language queries. Large Scale Movie Description Challenge (LSMDC) [26] consists of short clips from movies described by natural language.…”

Section: Related Workmentioning

confidence: 99%

“…(2) The descriptions are rich with over 100 words per paragraph. Figure 2 compares ActivityNet Caption [19] with the MSA dataset with examples. We can see that the descriptions in MSA are generally much richer and at a higher level, e.g.…”

Section: Msa Datasetmentioning

confidence: 99%

A Graph-Based Framework to Bridge Movies and Synopses

Xiong

Huang

Guo

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Inspired by the remarkable advances in video analytics, research teams are stepping towards a greater ambition -movie understanding. However, compared to those activity videos in conventional datasets, movies are significantly different. Generally, movies are much longer and consist of much richer temporal structures. More importantly, the interactions among characters play a central role in expressing the underlying story. To facilitate the efforts along this direction, we construct a dataset called Movie Synopses Associations (MSA) over 327 movies, which provides a synopsis for each movie, together with annotated associations between synopsis paragraphs and movie segments. On top of this dataset, we develop a framework to perform matching between movie segments and synopsis paragraphs. This framework integrates different aspects of a movie, including event dynamics and character interactions, and allows them to be matched with parsed paragraphs, based on a graph-based formulation. Our study shows that the proposed framework remarkably improves the matching accuracy over conventional featurebased methods. It also reveals the importance of narrative structures and character interactions in movie understanding. Dataset and code are available at: https:// ycxioooong.github.io/projects/moviesyn 1

show abstract

“…However, we are proposing a textual form of multimodal representation. There are two closely related tasks in the domain of machine learningvisual question-answering (VQA) [5] and dense video captioning [6] tasks. In these tasks, the model is trained to take in the video input and output a sequence of text that describes the video or answer questions.…”

Section: Textual Multimodal Representationmentioning

confidence: 99%

Detecting Depression in Dyadic Conversations with Multimodal Narratives and Visualizations

Kim

Yacef

2019

AI 2019: Advances in Artificial Intelligence

View full text Add to dashboard Cite

Conversations contain a wide spectrum of multimodal information that gives us hints about the emotions and moods of the speaker. In this paper, we developed a system that supports humans to analyze conversations. Our main contribution is the identification of appropriate multimodal features and the integration of such features into verbatim conversation transcripts. We demonstrate the ability of our system to take in a wide range of multimodal information and automatically generated a prediction score for the depression state of the individual. Our experiments showed that this approach yielded better performance than the baseline model. Furthermore, the multimodal narrative approach makes it easy to integrate learnings from other disciplines, such as conversational analysis and psychology. Lastly, this interdisciplinary and automated approach is a step towards emulating how practitioners record the course of treatment as well as emulating how conversational analysts have been analyzing conversations by hand.

show abstract

Dense-Captioning Events in Videos

Cited by 1,052 publications

References 66 publications

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

A Graph-Based Framework to Bridge Movies and Synopses

Detecting Depression in Dyadic Conversations with Multimodal Narratives and Visualizations

Contact Info

Product

Resources

About