Temporal Action Segmentation: An Analysis of Modern Techniques

Ding, Guodong; Sener, Fadime; Yao, Angela

doi:10.1109/tpami.2023.3327284

Cited by 28 publications

(6 citation statements)

References 138 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It involves analyzing a video sequence's temporal structure to identify and mark the boundaries between different actions. The aim is to accurately segment and label each action or activity, allowing for a more detailed understanding and analysis of the sequence [18].…”

Section: Temporal Action Segmentation (Tas)mentioning

confidence: 99%

“…Practically, TAS is implemented using precomputed frame-wise features as input because it avoids the more significant computational load required for learning video features [18]. In cases where the movement of objects is dynamic and there may be dependencies between video frames, a temporal or sequential model that uses a learning model such as a Convolutional Network, RNN, and Transformer is needed.…”

Section: Temporal Action Segmentation (Tas)mentioning

confidence: 99%

“…Pre-computed frame-wise features using optical flow have much cheaper computational costs compared to learning models (learning video features) [18], [19]. A method was introduced for the optical flow-based still image to video segmentation models, resulting in increased stability [20].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Temporal Action Segmentation in Sign Language System for Bahasa Indonesia (SIBI) Videos Using Optical Flow-Based Approach

I Dewa Made Bayu Atmaja Darmawan,

Linawati,

Sukadarmika

et al. 2024

Jurnal Ilmu Komputer dan Informasi

View full text Add to dashboard Cite

Sign language (SL) is vital in fostering communication for the deaf and hard-of-hearing communities. Continuous Sign Language Translation (CSLT) is a work that translates sign language into spoken language. CSLT translation is done by changing continuous forms into isolated signs. Segmenting morpheme signs from phrase signs has several challenges, such as the availability of annotated datasets and the complexity of continuous gesture movements. The Indonesian Sign Language (SIBI) system follows Indonesian grammatical norms, including word formation, in contrast to other sign languages with rules derived from their spoken language. In SIBI, a word can consist of a root word and an affix word. Therefore, temporal action segmentation in SIBI is important to reconstruct the results of translating each sign into spoken Indonesian sentences. This research uses an optical flow approach to segment temporal actions in SIBI videos. Optical flow methods that calculate changes in intensity between adjacent frames can be used to determine the occurrence of sign movement or vice versa to determine the delay between sign movements. The absence of intensity differences between the two frames indicates the boundary between sign gestures. This study tested the use of dense optical flow on videos containing SIBI sentences taken from 3 signers. Evaluation is done on several parameters in the dense optical flow algorithm, such as threshold size, PyrScale, and WinSize, to obtain the best accuracy. This paper shows that the optical flow algorithm successfully performs segmentation, as measured by Perf and F1r. The experimental results showed that the highest Perf and F1r yields were 0.8298 and 0.8524, respectively.

show abstract

Section: Temporal Action Segmentation (Tas)mentioning

confidence: 99%

Section: Temporal Action Segmentation (Tas)mentioning

confidence: 99%

See 1 more Smart Citation

Temporal Action Segmentation in Sign Language System for Bahasa Indonesia (SIBI) Videos Using Optical Flow-Based Approach

I Dewa Made Bayu Atmaja Darmawan,

Linawati,

Sukadarmika

et al. 2024

Jurnal Ilmu Komputer dan Informasi

View full text Add to dashboard Cite

show abstract

“…We further proposed a multi-stage structure with multi-level supervised contrastive loss to learn a well-structured embedding space for enhanced activity segmentation and recognition performance [33]. In parallel, action segmentation in video data has garnered considerable interest in both academia and industry [34]. MS-TCN [35] is a classic method in video action segmentation tasks, which achieves impressive segmentation results using dilated convolutions with residual connections.…”

Section: Fully Supervised Activity Segmentationmentioning

confidence: 99%

A Boundary Consistency-Aware Multitask Learning Framework for Joint Activity Segmentation and Recognition With Wearable Sensors

Xia

Chu

Pei

et al. 2023

IEEE Trans. Ind. Inf.

View full text Add to dashboard Cite

Human activity recognition (HAR) with wearables is one of the serviceable technologies in ubiquitous and mobile computing applications. The sliding-window scheme is widely adopted while suffering from the multi-class windows problem. As a result, there is a growing focus on joint segmentation and recognition with deep-learning methods, aiming at simultaneously dealing with HAR and time-series segmentation issues. However, obtaining the full activity annotations of wearable data sequences is resource-intensive or time-consuming, while unsupervised methods yield poor performance. To address these challenges, we propose a novel method for joint activity segmentation and recognition with timestamp supervision, in which only a single annotated sample is needed in each activity segment. However, the limited information of sparse annotations exacerbates the gap between recognition and segmentation tasks, leading to sub-optimal model performance. Therefore, the prototypes are estimated by class-activation maps to form a sample-to-prototype contrast module for well-structured embeddings. Moreover, with the optimal transport theory, our approach generates the sample-level pseudo-labels that take advantage of unlabeled data between timestamp annotations for further performance improvement. Comprehensive experiments on four public HAR datasets demonstrate that our model trained with timestamp supervision is superior to the state-of-the-art weakly-supervised methods and achieves comparable performance to the fully-supervised approaches.

show abstract

“…Traditional machine learning algorithms, that were dominant in human activity recognition until a few years ago [8,9], required extensive domain knowledge and time investment for feature engineering. In contrast, deep learning methods circumvent manual feature extraction through the hierarchical learning of complex data representations, although they demand significant data and computational resources [10].…”

Section: Introductionmentioning

confidence: 99%

Improving Time Study Methods Using Deep Learning-Based Action Segmentation Models

Gudlin,

Hegedić,

Golec

et al. 2024

Applied Sciences

View full text Add to dashboard Cite

In the quest for industrial efficiency, human performance within manufacturing systems remains pivotal. Traditional time study methods, reliant on direct observation and manual video analysis, are increasingly inadequate, given technological advancements. This research explores the automation of time study methods by deploying deep learning models for action segmentation, scrutinizing the efficacy of various architectural strategies. A dataset, featuring nine work activities performed by four subjects on three product types, was collected from a real manufacturing assembly process. Our methodology hinged on a two-step video processing framework, capturing activities from two perspectives: overhead and hand-focused. Through experimentation with 27 distinctive models varying in viewpoint, feature extraction method, and the architecture of the segmentation model, we identified improvements in temporal segmentation precision measured with the F1@IoU metric. Our findings highlight the limitations of basic Transformer models in action segmentation tasks, due to their lack of inductive bias and the limitations of a smaller dataset scale. Conversely, the 1D CNN and biLSTM architectures demonstrated proficiency in temporal data modeling, advocating for architectural adaptability over mere scale. The results contribute to the field by underscoring the interplay between model architecture, feature extraction method, and viewpoint integration in refining time study methodologies.

show abstract

Temporal Action Segmentation: An Analysis of Modern Techniques

Cited by 28 publications

References 138 publications

Temporal Action Segmentation in Sign Language System for Bahasa Indonesia (SIBI) Videos Using Optical Flow-Based Approach

Temporal Action Segmentation in Sign Language System for Bahasa Indonesia (SIBI) Videos Using Optical Flow-Based Approach

A Boundary Consistency-Aware Multitask Learning Framework for Joint Activity Segmentation and Recognition With Wearable Sensors

Improving Time Study Methods Using Deep Learning-Based Action Segmentation Models

Contact Info

Product

Resources

About