POS-Trends Dynamic-Aware Model for Video Caption

Wang, Lanxiao; Li, Hongliang; Qiu, Heqian; Wu, Qingbo; Meng, Fanman

doi:10.1109/tcsvt.2021.3131721

Cited by 18 publications

(3 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is achieved by generating correct words and providing direction for decoding. As shown by several MSVD, MSR-VTT, and VATEX tests, our approach is superior to the most recent and cutting-edge methods for solving BLEU-4, ROUGE-L, METEOR, and CIDEr [40].…”

Section: Literature Reviewmentioning

confidence: 80%

An Overview of Speech-To-Text Conversion

Aggarwal

2023

CIML

View full text Add to dashboard Cite

As a result of developments in science and technology, an automatic speech-to-text (STT) conversion system has been available. This system converts spoken words into text that can be read visually. People with trouble hearing may use this technology to communicate in other ways, including understanding voice communication and being able to follow directions using their visual abilities. There are instances when seeing something is more powerful than listening to something, particularly in long-distance communication; thus, speech-to-text conversion is crucial in situations like these. One of the fascinating developments to occur in the twenty-first century is the advent of machine learning. It has evolved from its roots in neurology studies conducted in the 1940s into something like artificial intelligence humans have created. Neural networks, a collection of complex structures, are the basis of machine learning. When combined with optimization techniques, these networks mimic the behaviour of neurons in the human brain and allow a computer to learn from its experiences. Here we explore one of many potential uses for such structures - the analysis of vocal performance in an original study. In particular, we dissect voice recognition systems to determine their inner workings.

show abstract

Section: Literature Reviewmentioning

confidence: 80%

An Overview of Speech-To-Text Conversion

Aggarwal

2023

CIML

View full text Add to dashboard Cite

show abstract

“…Researchers [78−85] usually use attention mechanisms in the temporal and spatial (regional) dimensions. Yao et al [86] tried to fuse different visual information in different frames based on the temporal attention module. Some methods [80,83] use a spatial attention mechanism to enhance the important parts within each frame.…”

Section: Methodsmentioning

confidence: 99%

A Survey of Vision and Language Related Multi-Modal Task

Wang¹,

Hu²,

Qiu³

et al. 2022

CAAI Artificial Intelligence Research

Self Cite

View full text Add to dashboard Cite

With the significant breakthrough in the research of single-modal related deep learning tasks, more and more works begin to focus on multi-modal tasks. Multi-modal tasks usually involve more than one different modalities, and a modality represents a type of behavior or state. Common multi-modal information includes vision, hearing, language, touch, and smell. Vision and language are two of the most common modalities in human daily life, and many typical multi-modal tasks focus on these two modalities, such as visual captioning and visual grounding. In this paper, we conduct in-depth research on typical tasks of vision and language from the perspectives of generation, analysis, and reasoning. First, the analysis and summary with the typical tasks and some pretty classical methods are introduced, which will be generalized from the aspects of different algorithmic concerns, and be further discussed frequently used datasets and metrics. Then, some other variant tasks and cutting-edge tasks are briefly summarized to build a more comprehensive vision and language related multi-modal tasks framework. Finally, we further discuss the development of pre-training related research and make an outlook for future research. We hope this survey can help relevant researchers to understand the latest progress, existing problems, and exploration directions of vision and language multi-modal related tasks, and provide guidance for future research.

show abstract

“…Next, each color channel was bandpass filtered using a Butterworth filter (0.75-2.75 Hz; 6th order). We then dimension-reduced the RGB array, initially representing changes in average 3D RGB color space, to a single array, eventually representing only heart-beat-related fluctuations, using the plane-orthogonal-to-skin algorithm (POS; Wang, Li et al 2022)) with a sliding window size of 1.6 s. This final array was converted to the frequency domain using a time-frequency analysis based on Lomb-Scargle periodogram calculations per sliding window of 10 s with a temporal resolution of 240 points and frequency resolution of 120 points. The resulting power density functions, represented in a 2D (240 by 120) data array, were converted to signal-to-noise ratios (i.e., SNR; also termed coherence) per time point by dividing each power value by the sum of all absolute power values.…”

Section: Rppg Extractionmentioning

confidence: 99%

Remote photoplethysmography (rPPG) in the wild: Remote heart rate imaging via online webcams.

Finotti¹,

Lernia²,

Tsakiris³

et al. 2022

Preprint

View full text Add to dashboard Cite

Remote photoplethysmography is a low-cost technique to measure physiological parameters such as the heartrate by analysing videos of a person. There has been growing attention in this technique due to the increased possibilities and demand for running psychological experiments in online platforms. Many progresses have been made due to technological advancements in commercially available cameras and in the algorithms used to process the videos. Despite these advancements, past research has shown that suboptimal conditions of the video recordings can severely impair the accuracy of the rPPG. Our aim was to test the accuracy of a modified open-source rPPG algorithm on videos collected via an online platform, without control on the hardware of the participants and the contextual variables, such as illumination, distance and motion. Across two experiments, we compared the results of the rPPG extraction algorithm to a validated dataset used for rPPG testing, and then we collected 231 online video recordings and compared the results of the rPPG extraction against a validated mobile heart rate application. Results showed that the rPPG was highly accurate on both datasets thus providing an improved tool for recording and analyzing heartrate in online experiments.

show abstract

POS-Trends Dynamic-Aware Model for Video Caption

Cited by 18 publications

References 61 publications

An Overview of Speech-To-Text Conversion

An Overview of Speech-To-Text Conversion

A Survey of Vision and Language Related Multi-Modal Task

Remote photoplethysmography (rPPG) in the wild: Remote heart rate imaging via online webcams.

Contact Info

Product

Resources

About