Structural-RNN: Deep Learning on Spatio-Temporal Graphs

Jain, Anurag; Zamir, Amir; Savarese, Silvio; Saxena, Ashutosh

doi:10.1109/cvpr.2016.573

Cited by 1,017 publications

(989 citation statements)

References 56 publications

Supporting

Mentioning

982

Contrasting

Order By: Relevance

“…"Deep" neural networks with many sequential hidden layers also possess desirable theoretical properties for function representation (Lu et al 2017). The field of deep learning, which applies such networks, provides a variety of network architectures to account for temporal structure (Hochreiter & Schmidhuber 1997), spatial structure on regular grids (Long et al 2015), or graphs (Niepert et al 2016), sets of unordered irregular points (Li et al 2018), and spatiotemporal data on grids or graphs (Xingjian et al 2015;Jain et al 2016).…”

Section: Related Work Neural Networkmentioning

confidence: 99%

Neural hierarchical models of ecological populations

Joseph

2019

Preprint

View full text Add to dashboard Cite

Neural networks are increasingly being used in science to infer hidden dynamics of natural systems from noisy observations, a task typically handled by hierarchical models in ecology. This paper describes an emergent class of hierarchical models parameterized by neural networks -neural hierarchical models -and develops a case study with bird populations in the United States. The derivation of such models is outlined beginning with the relationship between linear regression and neural networks, and analogously between a hierarchical ecological model (a dynamic occupancy model), and its neural counterpart. The resulting neural hierarchical model, trained on millions of detection/non-detection time series for hundreds of species, provides insights into colonization and extinction at a continental scale. The model also demonstrates good quantitative predictive performance and reasonable qualitative results about range shifts and macroecological patterns of population persistence at range boundaries. Flexible models are increasingly needed that scale to large data and represent ecological processes. Neural hierarchical models satisfy this need, providing a bridge between deep learning and ecological modeling that combines the function representation power of neural networks with the inferential capacity of hierarchical models.

show abstract

Section: Related Work Neural Networkmentioning

confidence: 99%

Neural hierarchical models of ecological populations

Joseph

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Jain et al [28] developed a novel NN architecture that introduces spatio-temporal graphs in its structure. More specifically, the factor components in the st-graphs are grouped and modeled with RNNs.…”

Section: Nn For Motion Modelingmentioning

confidence: 99%

Mathematical Modeling and Evaluation of Human Motions in Physical Therapy Using Mixture Density Neural Networks

Vakanski¹,

Ferguson²,

S³

2016

Physiother Rehabil

View full text Add to dashboard Cite

Objective-The objective of the proposed research is to develop a methodology for modeling and evaluation of human motions, which will potentially benefit patients undertaking a physical rehabilitation therapy (e.g., following a stroke or due to other medical conditions). The ultimate aim is to allow patients to perform home-based rehabilitation exercises using a sensory system for capturing the motions, where an algorithm will retrieve the trajectories of a patient's exercises, will perform data analysis by comparing the performed motions to a reference model of prescribed motions, and will send the analysis results to the patient's physician with recommendations for improvement.Methods-The modeling approach employs an artificial neural network, consisting of layers of recurrent neuron units and layers of neuron units for estimating a mixture density function over the spatio-temporal dependencies within the human motion sequences. Input data are sequences of motions related to a prescribed exercise by a physiotherapist to a patient, and recorded with a motion capture system. An autoencoder subnet is employed for reducing the dimensionality of captured sequences of human motions, complemented with a mixture density subnet for probabilistic modeling of the motion data using a mixture of Gaussian distributions.Results-The proposed neural network architecture produced a model for sets of human motions represented with a mixture of Gaussian density functions. The mean log-likelihood of observed sequences was employed as a performance metric in evaluating the consistency of a subject's performance relative to the reference dataset of motions. A publically available dataset of human motions captured with Microsoft Kinect was used for validation of the proposed method. Conclusion-The article presents a novel approach for modeling and evaluation of human motions with a potential application in home-based physical therapy and rehabilitation. The described approach employs the recent progress in the field of machine learning and neural networks in developing a parametric model of human motions, by exploiting the representational power of these algorithms to encode nonlinear input-output dependencies over long temporal horizons.

show abstract

“…Despite * The first two authors contributed equally. recent progress in data-driven modelling of human motion [7,8,14,20,25,33], this task remains difficult for machines. The difficulty of the task is manifold.…”

Section: Introductionmentioning

confidence: 99%

“…However, these are hard to model algorithmically due to i) the inter-dependencies between joints and ii) the influence of high-level activities on the motion sequences (e.g., transition from walking to jumping). In fact many recent approaches forgo explicit modelling of human motion [14] in favor of pure data-driven models [8,20,25].…”

Section: Introductionmentioning

confidence: 99%

Structured Prediction Helps 3D Human Motion Modelling

Aksan

Kaufmann

Hilliges

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

197

151

View full text Add to dashboard Cite

Figure 1: We introduce a structured prediction layer (SPL) to the task of 3D human motion modelling. The SP-layer explicitly decomposes the pose into individual joints and can be interfaced with a variety of baseline architectures. We show that on H3.6M and a recent, much larger dataset, AMASS, a variety of baseline models benefit when augmented with an SP-layer. AbstractHuman motion prediction is a challenging and important task in many computer vision application domains. Existing work only implicitly models the spatial structure of the human skeleton. In this paper, we propose a novel approach that decomposes the prediction into individual joints by means of a structured prediction layer that explicitly models the joint dependencies. This is implemented via a hierarchy of small-sized neural networks connected analogously to the kinematic chains in the human body as well as a joint-wise decomposition in the loss function. The proposed layer is agnostic to the underlying network and can be used with existing architectures for motion modelling. Prior work typically leverages the H3.6M dataset. We show that some state-of-the-art techniques do not perform well when trained and tested on AMASS, a recently released dataset 14 times the size of H3.6M. Our experiments indicate that the proposed layer increases the performance of motion forecasting irrespective of the base network, jointangle representation, and prediction horizon. We furthermore show that the layer also improves motion predictions qualitatively. We make code and models publicly available at https://ait.ethz.ch/projects/2019/spl.

show abstract

Structural-RNN: Deep Learning on Spatio-Temporal Graphs

Cited by 1,017 publications

References 56 publications

Neural hierarchical models of ecological populations

Neural hierarchical models of ecological populations

Mathematical Modeling and Evaluation of Human Motions in Physical Therapy Using Mixture Density Neural Networks

Structured Prediction Helps 3D Human Motion Modelling

Contact Info

Product

Resources

About