HAKE: A Knowledge Engine Foundation for Human Activity Understanding

Abstract

Human activity understanding is of widespread interest in artificial intelligence and spans diverse applications like health care and behavior analysis. Although there have been advances with deep learning, it remains challenging. The object recognition-like solutions usually try to map pixels to semantics directly, but activity patterns are much different from object patterns, thus hindering another success. In this work, we propose a novel paradigm to reformulate this task in two-stage: first mapping pixels to an intermediate space spanned by atomic activity primitives, then programming detected primitives with interpretable logic rules to infer semantics. To afford a representative primitive space, we build a knowledge base including 26+ M primitive labels and logic rules from human priors or automatic discovering. Our framework, Human Activity Knowledge Engine (HAKE), exhibits superior generalization ability and performance upon canonical methods on challenging benchmarks. Code and data are available at http://hake-mvig.cn/.

Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detection

“…3, we evaluate how HOI learning can benefit from the interactiveness detection results of our method. Here we use instance-level supervision without annotations from HAKE [21,19] for interactiveness learning to compare with TIN++ [17]. In Tab.…”