HalluciNet-ing Spatiotemporal Representations Using a 2D-CNN

Abstract

Spatiotemporal representations learned using 3D convolutional neural networks (CNN) are currently used in state-of-the-art approaches for action-related tasks. However, 3D-CNN are notorious for being memory and compute resource intensive as compared with more simple 2D-CNN architectures. We propose to hallucinate spatiotemporal representations from a 3D-CNN teacher with a 2D-CNN student. By requiring the 2D-CNN to predict the future and intuit upcoming activity, it is encouraged to gain a deeper understanding of actions and how they evolve. The hallucination task is treated as an auxiliary task, which can be used with any other action-related task in a multitask learning setting. Thorough experimental evaluation, it is shown that the hallucination task indeed helps improve performance on action recognition, action quality assessment, and dynamic scene recognition tasks. From a practical standpoint, being able to hallucinate spatiotemporal representations without an actual 3D-CNN can enable deployment in resource-constrained scenarios, such as with limited computing power and/or lower bandwidth. We also observed that our hallucination task has utility not only during the training phase, but also during the pre-training phase.

Computer Vision Algorithms of DigitSeis for Building a Vectorised Dataset of Historical Seismograms from the Archive of Royal Observatory of Belgium

“…As a response to these needs, various programs for automatic, quick and accurate digitising and modelling of seismograms were developed. The advances in pattern recognition are supported by rapid growth of programming languages [ 75 , 76 , 77 , 78 , 79 ], and specifically, Python [ 80 , 81 , 82 ], improvements of algorithms of ML for image segmentation and clustering [ 83 , 84 , 85 , 86 , 87 ], and signal processing [ 88 , 89 ].…”