Humans Predict Action using Grammar-like Structures

Wörgötter, Florentin; Ziaeetabar, Fatemeh; Pfeiffer, Stefan; Kaya, Ozan; Kulvičius, Tomas; Tamošiūnaitė, Minija

doi:10.1038/s41598-020-60923-5

Cited by 9 publications

(4 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that all methodological details concerning our spatial relations definition (section 5.4.1) and their computation (section 5.5) as well as details of the similarity measurement algorithm (section 5.6) were reported previously in [23] and [28]. Hence, the next three subsections are essentially a repetition from those two papers without many changes.…”

Section: Details Of Machine Action Predictionmentioning

confidence: 98%

“…The approach was developed based on previous assumptions on the importance of spatial relations in action recognition [24][25][26][27][28] and stands in contrast to action recognition and prediction methods based on time continuous information, like trajectories [29][30][31][32] or continuous action videos [33][34][35]. It also stands in contrast to the methods exploiting rich contextual information [36][37][38][39][40].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Using enriched semantic event chains to model human action prediction based on (minimal) spatial information

et al. 2020

View full text Add to dashboard Cite

Predicting other people’s upcoming action is key to successful social interactions. Previous studies have started to disentangle the various sources of information that action observers exploit, including objects, movements, contextual cues and features regarding the acting person’s identity. We here focus on the role of static and dynamic inter-object spatial relations that change during an action. We designed a virtual reality setup and tested recognition speed for ten different manipulation actions. Importantly, all objects had been abstracted by emulating them with cubes such that participants could not infer an action using object information. Instead, participants had to rely only on the limited information that comes from the changes in the spatial relations between the cubes. In spite of these constraints, participants were able to predict actions in, on average, less than 64% of the action’s duration. Furthermore, we employed a computational model, the so-called enriched Semantic Event Chain (eSEC), which incorporates the information of different types of spatial relations: (a) objects’ touching/untouching, (b) static spatial relations between objects and (c) dynamic spatial relations between objects during an action. Assuming the eSEC as an underlying model, we show, using information theoretical analysis, that humans mostly rely on a mixed-cue strategy when predicting actions. Machine-based action prediction is able to produce faster decisions based on individual cues. We argue that human strategy, though slower, may be particularly beneficial for prediction of natural and more complex actions with more variable or partial sources of information. Our findings contribute to the understanding of how individuals afford inferring observed actions’ goals even before full goal accomplishment, and may open new avenues for building robots for conflict-free human-robot cooperation.

show abstract

Section: Details Of Machine Action Predictionmentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

Using enriched semantic event chains to model human action prediction based on (minimal) spatial information

et al. 2020

View full text Add to dashboard Cite

show abstract

“…We trained our algorithm on 7 different trajectories performing ordinary robotic tasks. The tasks are hide, unhide, move down, move up, pick and place, put on top and take down (see e.g., Wörgötter et al (2020)). Movement data is given in 3 dimensional Cartesian coordinates, resulting in three outputs or -biologically speaking -in three rate coded output neurons.…”

Section: Stability and Output Learningmentioning

confidence: 99%

Robust Trajectory Generation for Robotic Control on the Neuromorphic Research Chip Loihi

2020

View full text Add to dashboard Cite

Neuromorphic hardware has several promising advantages compared to von Neumann architectures and is highly interesting for robot control. However, despite the high speed and energy efficiency of neuromorphic computing, algorithms utilizing this hardware in control scenarios are still rare. One problem is the transition from fast spiking activity on the hardware, which acts on a timescale of a few milliseconds, to a control-relevant timescale on the order of hundreds of milliseconds. Another problem is the execution of complex trajectories, which requires spiking activity to contain sufficient variability, while at the same time, for reliable performance, network dynamics must be adequately robust against noise. In this study we exploit a recently developed biologically-inspired spiking neural network model, the so-called anisotropic network. We identified and transferred the core principles of the anisotropic network to neuromorphic hardware using Intel's neuromorphic research chip Loihi and validated the system on trajectories from a motor-control task performed by a robot arm. We developed a network architecture including the anisotropic network and a pooling layer which allows fast spike read-out from the chip and performs an inherent regularization. With this, we show that the anisotropic network on Loihi reliably encodes sequential patterns of neural activity, each representing a robotic action, and that the patterns allow the generation of multidimensional trajectories on control-relevant timescales. Taken together, our study presents a new algorithm that allows the generation of complex robotic movements as a building block for robotic control using state of the art neuromorphic hardware.

show abstract

“…The tasks are hide, unhide, move down, move up, pick and place, put on top and take down (see e.g. Wörgötter et al, 2020). Movement data is given in 3 dimensional Cartesian coordinates, resulting in three outputs orbiologically speaking -in three rate coded output neurons.…”

Section: Stability and Output Learningmentioning

confidence: 99%

Robust trajectory generation for robotic control on the neuromorphic research chip Loihi

Michaelis,

Lehr,

Tetzlaff

2020

Preprint

View full text Add to dashboard Cite

Neuromorphic hardware has several promising advantages compared to von Neumann architectures and is highly interesting for robot control. However, despite the high speed and energy efficiency of neuromorphic computing, algorithms utilizing this hardware in control scenarios are still missing. One problem is the transition from fast spiking activity on the hardware, which acts on a timescale of a few milliseconds, to a control-relevant timescale on the order of hundreds of milliseconds. Another problem is to enable the execution of complex trajectories, requiring the spiking activity to contain sufficient variability, while at the same time, for reliable performance, network dynamics require adequate robustness against noise. In this study we exploit a recently developed biologically-inspired spiking neural network model, the so-called anisotropic network, as the basis for a neuromorphic algorithm for robotic control. For this, we identified and transferred the core principles of the anisotropic network to neuromorphic hardware using Intel's neuromorphic research chip Loihi and validated the system on trajectories from a motor-control task performed by a robot arm. We show that the anisotropic network on Loihi reliably encodes sequential patterns of neural activity, each representing a robotic action, and that the patterns allow the generation of multidimensional trajectories on control-relevant timescales. Taken together, our study presents a new algorithm that allows the control of complex robotic movements using state of the art neuromorphic hardware.

show abstract

Humans Predict Action using Grammar-like Structures

Cited by 9 publications

References 32 publications

Using enriched semantic event chains to model human action prediction based on (minimal) spatial information

Using enriched semantic event chains to model human action prediction based on (minimal) spatial information

Robust Trajectory Generation for Robotic Control on the Neuromorphic Research Chip Loihi

Robust trajectory generation for robotic control on the neuromorphic research chip Loihi

Contact Info

Product

Resources

About