Multi-label Class-imbalanced Action Recognition in Hockey Videos via 3D Convolutional Neural Networks

Abstract

Automatic analysis of the video is one of most complex problems in the fields of computer vision and machine learning. A significant part of this research deals with (human) activity recognition (HAR) since humans, and the activities that they perform, generate most of the video semantics. Video-based HAR has applications in various domains, but one of the most important and challenging is HAR in sports videos. Some of the major issues include high inter-and intra-class variations, large class imbalance, the presence of both group actions and single player actions, and recognizing simultaneous actions, i.e., the multi-label learning problem. Keeping in mind these challenges and the recent success of CNNs in solving various computer vision problems, in this work, we implement a 3D CNN based multilabel deep HAR system for multi-label class-imbalanced action recognition in hockey videos. We test our system for two different scenarios: an ensemble of k binary networks vs. a single k-output network, on a publicly available dataset. We also compare our results with the system that was originally designed for the chosen dataset. Experimental results show that the proposed approach performs better than the existing solution.

Scene Classification for Sports Video Summarization Using Transfer Learning

“…Video summarized using the proposed system received a good score of 4 out of 5 from the participants. Sozykin K et al [17] proposed a 3D CNN-based multi-label deep Human Action Recognition (HAR) system for sports video summarization for the sport of Hockey and presented more than ten classes. Data pre-processing techniques like resizing, normalization, windowing, and sequence labeling were used.…”

Deep learning in sport video analysis: a review

“…They used pre-trained CNN to first extract the features then use LSTM for classification of the five types of events which are dump in, dump out, loose puck recovery, pass and shot. Sozykin et al [35] presented a 3D CNN based action recognition system for multi-class imbalanced in ice hockey. They first extract features from both single image and a slice of frames using CNN.…”

Simultaneous and Spatiotemporal Detection of Different Levels of Activity in Multidimensional Data

“…The use of multiple labels to represent multiple actions has grown in popularity due to the interest in detecting and recognizing simultaneous activity in videos. For example, concurrent action recognition in hockey videos [1] could indicate that a 'Play', 'Face Off' and 'Fight' took place at the same time. Similarly, the ability to tag multiple facial expressions in videos can be accomplished using multilabels to detect emotions, a crucial component of HCI [2].…”

“…Similarly, the ability to tag multiple facial expressions in videos can be accomplished using multilabels to detect emotions, a crucial component of HCI [2]. However, in all of the aforementioned references, an action or a combination of actions is assigned to a label, which in turn is given a binary assignment representing its absence (0) or its presence (1).…”

“…The contributions of the proposed work include the following: i) we incorporate a novel spatial aspect to our multilabel by assigning each label to a different region of interest. To the best of our knowledge, this is a unique way of handling the description of concurrent actions within a spatial context; ii) our proposed approach is able to detect different levels of activity, in contrast to existing approaches providing only a binary result as the presence or absence indicator [1], [2], [12]. Instead of simply assigning a 0 (absence of motion) or a 1 (presence of motion) to each label, we can assign, without loss of generality, a 0 (absence of motion), 1 (low level of motion) or 2 (high level of motion) to describe the level of motion in different regions of interest at a certain moment in time.…”