Sequential Interval Network for parsing complex structured activity

Vo, Nam; Bobick, Aaron F.

doi:10.1016/j.cviu.2015.07.006

Cited by 6 publications

(5 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A similar effort by Vo and Bobick [8] used hand-defined probabilistic context-free grammars (PCFGs) to instantiate a graphical model for the purpose of segmenting and classifying compositional action sequences. They evaluated their method on videos of toy airplane construction, along with cooking and human activity datasets.…”

Section: A Activity Recognitionmentioning

confidence: 99%

“…In assembly processes, these kinematic changes take the form of elementary actions that incorporate parts into a structure or remove them from it. Previous assembly action recognition representations were essentially of the form add(part) [8] or MERGE(part 1 , part 2 ) [6]. We extend this representation to apply to general assembly scenarios by observing that an assembly action can be thought of as a difference between two states: the current configuration of a spatial assembly and the new configuration that is produced by the action.…”

Section: B Assembly Sequencesmentioning

confidence: 99%

“…We use the Adam optimizer with a learning rate of 0.001. Our network has 6 layers with the following number of channels: [8,8,16,16,32,32], and kernel size 25. We use a dropout rate of 0.2.…”

Section: B Toy Blocksmentioning

confidence: 99%

See 2 more Smart Citations

Fine-grained activity recognition for assembly videos

Jones¹,

Cortesa²,

Shelton³

et al. 2020

Preprint

View full text Add to dashboard Cite

In this paper we address the task of recognizing assembly actions as a structure (e.g. a piece of furniture or a toy block tower) is built up from a set of primitive objects. Recognizing the full range of assembly actions requires perception at a level of spatial detail that has not been attempted in the action recognition literature to date. We extend the fine-grained activity recognition setting to address the task of assembly action recognition in its full generality by unifying assembly actions and kinematic structures within a single framework. We use this framework to develop a general method for recognizing assembly actions from observation sequences, along with observation features that take advantage of a spatial assembly's special structure. Finally, we evaluate our method empirically on two application-driven data sources:(1) An IKEA furniture-assembly dataset, and (2) A blockbuilding dataset. On the first, our system recognizes assembly actions with an average framewise accuracy of 70% and an average normalized edit distance of 10%. On the second, which requires fine-grained geometric reasoning to distinguish between assemblies, our system attains an average normalized edit distance of 23%-a relative improvement of 69% over prior work.

show abstract

Section: A Activity Recognitionmentioning

confidence: 99%

Section: B Assembly Sequencesmentioning

confidence: 99%

See 1 more Smart Citation

Fine-grained activity recognition for assembly videos

Jones¹,

Cortesa²,

Shelton³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Ryoo and Aggarwal (2009) developed an application that recognized high-level hierarchical human activities with CFG. Vo and Bobick (2016) introduced sequential interval network (SIN) to differentiate complex human activities. The method based on CFG was implemented to toy assembly and action recognition.…”

Section: Related Workmentioning

confidence: 99%

An approach for learning from robots using formal languages and automata

Aslan

Durdu

Sabancı

et al. 2019

View full text Add to dashboard Cite

Purpose In this study, human activity with finite and specific ranking is modeled with finite state machine, and an application for human–robot interaction was realized. A robot arm was designed that makes specific movements. The purpose of this paper is to create a language associated to a complex task, which was then used to teach individuals by the robot that knows the language. Design/methodology/approach Although the complex task is known by the robot, it is not known by the human. When the application is started, the robot continuously checks the specific task performed by the human. To carry out the control, the human hand is tracked. For this, the image processing techniques and the particle filter (PF) based on the Bayesian tracking method are used. To determine the complex task performed by the human, the task is divided into a series of sub-tasks. To identify the sequence of the sub-tasks, a push-down automata that uses a context-free grammar language structure is developed. Depending on the correctness of the sequence of the sub-tasks performed by humans, the robot produces different outputs. Findings This application was carried out for 15 individuals. In total, 11 out of the 15 individuals completed the complex task correctly by following the different outputs. Originality/value This type of study is suitable for applications to improve human intelligence and to enable people to learn quickly. Also, the risky tasks of a person working in a production or assembly line can be controlled with such applications by the robots.

show abstract

“…In addition to the dedicated effort made for modeling the temporal structures of video frames, some prior works attempted to model the temporal dependency of action labels [39][40][41]. They trained their classification models in presegmented videos and created grammars or statistical n-gram language models of action labels, to guide their classification models for action detection and parsing.…”

Section: Related Work 21 Action Learning From Pre-segmentedmentioning

confidence: 99%

CTC Network with Statistical Language Modeling for Action Sequence Recognition in Videos

Lin

Inoue

Sairyo

2017

Proceedings of the on Thematic Workshops of ACM Multimedia 2017

View full text Add to dashboard Cite

We propose a method for recognizing an action sequence in which several actions are concatenated and their boundaries are not given. The proposed method combines Connectionist Temporal Classification (CTC) and a statistical language model. CTC can learn the nature of each element action given no boundary information in an end-to-end manner. The statistical language model can learn the relationship between actions. We evaluate our method on the Breakfast dataset. When we use the trigram as the language model, its accuracy rate is 43.4%, which is better than the state-of-theart ECTC method by 6.7 percentage points.

show abstract

Sequential Interval Network for parsing complex structured activity

Cited by 6 publications

References 27 publications

Fine-grained activity recognition for assembly videos

Fine-grained activity recognition for assembly videos

An approach for learning from robots using formal languages and automata

CTC Network with Statistical Language Modeling for Action Sequence Recognition in Videos

Contact Info

Product

Resources

About