From pose to activity: Surveying datasets and introducing CONVERSE

Abstract

We present a review on the current state of publicly available datasets within the human action recognition community; highlighting the revival of pose based methods and recent progress of understanding person-person interaction modeling. We categorize datasets regarding several key properties for usage as a benchmark dataset; including the number of class labels, ground truths provided, and application domain they occupy. We also consider the level of abstraction of each dataset; grouping those that present actions, interactions and higher level semantic activities. The survey identifies key appearance and pose based datasets, noting a tendency for simplistic, emphasized, or scripted action classes that are often readily definable by a stable collection of sub-action gestures. There is a clear lack of datasets that provide closely related actions, those that are not implicitly identified via a series of poses and gestures, but rather a dynamic set of interactions. We therefore propose a novel dataset that represents complex conversational interactions between two individuals via 3D pose. 8 pairwise interactions describing 7 separate conversation based scenarios were collected using two Kinect depth sensors. The intention is to provide events that are constructed from numerous primitive actions, interactions and motions, over a period of time; providing a set of subtle action classes that are more representative of the real world, and a challenge to currently developed recognition methodologies. We believe this is among one of the first datasets devoted to conversational interaction classification using 3D pose features and the attributed papers show this task is indeed possible. The full dataset is made publicly available to the research community at [1].

Vision-based human activity recognition: a survey

“…Surveys on dataset benchmarks for human action recognition from visual data constitute another field of research tackled in [28,44,61,66,98,181]. They aim to guide researchers in the selection of the most suitable dataset for benchmarking their algorithms.…”

“…In addition, [66] presents a summary of the results obtained on the recent ASLAN benchmark [79], which was designed to reflect on the variety of challenges that modern activity recognition systems are expected to overcome. Authors in [44] propose a novel dataset, called CONVERSE, that represents complex conversational interactions between two individuals via 3D pose. Similarly, authors in [20,46,180,181,232] present a set of comprehensive reviews of the most commonly used RGB-D video-based activity recognition datasets.…”

Didn't see that coming: a survey on non-verbal social human behavior forecasting

“…Usually, third-view datasets consist of structured interactions where participants need to follow basic directives which favor spontaneous and fluent interactions. Despite the fact that conversations are the most common interaction structure, there are datasets which aim at fostering specific social signals like leadership, competitiveness, empathy, or affect, and therefore engage the participants in competitive/cooperative scenarios (Hung and Chittaranjan, 2010;Sanchez-Cortes et al, 2012;Rehg et al, 2013;Ringeval et al, 2013;Vella and Paggio, 2013;Bambach et al, 2015;Salter et al, 2015;Edwards et al, 2016;Beyan et al, 2016;Georgakis et al, 2017;Doyran et al, 2021;. Other datasets, instead, record in-the-wild interactions during the so-called cocktail parties (Alameda-Pineda et al, 2016;Cabrera-Quiros et al, 2018) and represent very interesting benchmarks to study group dynamics.…”

Labeling subtle conversational interactions within the CONVERSE dataset

“…In reality however, it can often be that quite subtle interactions are composed of numerous small gestures and interactions over a long period of time. The CONVERSE dataset therefore introduces a problem that utilizes pose based information to represent subtle and complex interaction classes that are not readily definable by the poses they contain [20]. The conversational classes shown are common interactions in daily life; however they do not exhibit an explicit relationship to the pose of the individual at any given frame.…”