Trained on large datasets, deep learning (DL) can accurately classify videos into hundreds of diverse classes. However, video data is expensive to annotate. Zero-shot learning (ZSL) proposes one solution to this problem. ZSL trains a model once, and generalizes to new tasks whose classes are not present in the training dataset. We propose the first end-to-end algorithm for ZSL in video classification. Our training procedure builds on insights from recent video classification literature and uses a trainable 3D CNN to learn the visual features. This is in contrast to previous video ZSL methods, which use pretrained feature extractors. We also extend the current benchmarking paradigm: Previous techniques aim to make the test task unknown at training time but fall short of this goal. We encourage domain shift across training and test data and disallow tailoring a ZSL model to a specific test dataset. We outperform the state-of-the-art by a wide margin. Our code, evaluation procedure and model weights are available at github.com/bbrattoli/ZeroShotVideoClassification. * Work done during an internship at Amazon.

Current neuromodulatory strategies to enhance motor recovery after stroke often target large brain areas non-specifically and without sufficient understanding of their interaction with internal repair mechanisms. Here we developed a novel therapeutic approach by specifically activating corticospinal circuitry using optogenetics after large strokes in rats. Similar to a neuronal growth-promoting immunotherapy, optogenetic stimulation together with intense, scheduled rehabilitation leads to the restoration of lost movement patterns rather than induced compensatory actions, as revealed by a computer vision-based automatic behavior analysis. Optogenetically activated corticospinal neurons promote axonal sprouting from the intact to the denervated cervical hemi-cord. Conversely, optogenetically silencing subsets of corticospinal neurons in recovered animals, results in mistargeting of the restored grasping function, thus identifying the reestablishment of specific and anatomically localized cortical microcircuits. These results provide a conceptual framework to improve established clinical techniques such as transcranial magnetic or transcranial direct current stimulation in stroke patients.

In this paper we present a self-supervised method for representation learning utilizing two different modalities. Based on the observation that cross-modal information has a high semantic meaning we propose a method to effectively exploit this signal. For our approach we utilize video data since it is available on a large scale and provides easily accessible modalities given by RGB and optical flow. We demonstrate state-of-the-art performance on highly contested action recognition datasets in the context of self-supervised learning. We show that our feature representation also transfers to other tasks and conduct extensive ablation studies to validate our core contributions. Code and model can be found at https://github.com/nawidsayed/Cross-and-Learn.

Self-supervised learning of convolutional neural networks can harness large amounts of cheap unlabeled data to train powerful feature representations. As surrogate task, we jointly address ordering of visual data in the spatial and temporal domain. The permutations of training samples, which are at the core of self-supervision by ordering, have so far been sampled randomly from a fixed preselected set. Based on deep reinforcement learning we propose a sampling policy that adapts to the state of the network, which is being trained. Therefore, new permutations are sampled according to their expected utility for updating the convolutional feature representation. Experimental evaluation on unsupervised and transfer learning tasks demonstrates competitive performance on standard benchmarks for image and video classification and nearest neighbor retrieval.