Spatiotemporal Similarity Search in 3D Motion Capture Gesture Streams

Beecks, Christian; Hassani, Marwan; Hinnell, Jennifer; Schüller, Daniel; Brenger, Bela; Mittelberg, Irene; Seidl, Thomas

doi:10.1007/978-3-319-22363-6_19

Cited by 13 publications

(8 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The strengths and weaknesses of the feature signature model have been investigated in various works [35][36][37] and its applicability reaches for instance from similarity search in multimedia databases [38][39][40] to scientific databases. 41,42 4 Methods…”

Section: Related Workmentioning

confidence: 99%

Content-based retrieval in videos from laparoscopic surgery

et al. 2016

Self Cite

View full text Add to dashboard Cite

In the field of medical endoscopy more and more surgeons are changing over to record and store videos of their endoscopic procedures for long-term archival. These endoscopic videos are a good source of information for explanations to patients and follow-up operations. As the endoscope is the "eye of the surgeon", the video shows the same information the surgeon has seen during the operation, and can describe the situation inside the patient much more precisely than an operation report would do. Recorded endoscopic videos can also be used for training young surgeons and in some countries the long-term archival of video recordings from endoscopic procedures is even enforced by law. A major challenge, however, is to efficiently access these very large video archives for later purposes. One problem, for example, is to locate specific images in the videos that show important situations, which are additionally captured as static images during the procedure. This work addresses this problem and focuses on contentbased video retrieval in data from laparoscopic surgery. We propose to use feature signatures, which can appropriately and concisely describe the content of laparoscopic images, and show that by using this content descriptor with an appropriate metric, we are able to efficiently perform content-based retrieval in laparoscopic videos. In a dataset with 600 captured static images from 33 hours recordings, we are able to find the correct video segment for more than 88% of these images.Feature signatures, as for instance investigated in the work of Beecks, 1 are a compact yet efficient way of describing the content of an image. They are obtained by clustering characteristic image features into a compact feature representation that adapts to individual image contents. In this paper, we propose to utilize feature signatures over a 7-dimensional feature space including position, color, and texture information for content-based retrieval in laparoscopic videos. To this end, we investigate different distance-based similarity measures including the Earth Mover's Distance, 2 the Signature Quadratic Form Distance, 3, 4 and the Signature Matching Distance. 5 Through a large evaluation with video data from 33 hours of laparoscopic surgery we show that the proposed signature-based approaches are able to efficiently perform content-based retrieval in laparoscopic videos. Description of PurposeIn endoscopic surgery, operating surgeons usually record video segments and capture static images during the operation, which they often need after the procedure. 6 However, while the static images can be used for later discussions with patients and colleagues, and can be easily integrated into operation reports, they lack of a complete and precise information, such as movements of operation instruments and the dynamics of performed actions. The recorded videos, on the other hand, contain all these important details, since they show exactly the same images that were visible on the monitors in the operation room, which were also used by the...

show abstract

Section: Related Workmentioning

confidence: 99%

Content-based retrieval in videos from laparoscopic surgery

et al. 2016

Self Cite

View full text Add to dashboard Cite

show abstract

“…It is clear, therefore, that if the laborious task of hand coding gesture kinematics can be replaced with reliable and objective automated methods, considerable advantages will accrue for researchers. In the present article, we contribute to this joint effort by introducing and validating novel methods for studying gesture-speech dynamics (e.g., Beecks et al, 2015;Danner et al, 2018;Krivokapić et al, 2017;Schueller et al, 2017;Pouw & Dixon 2019b), and by providing a tutorial for overcoming some common challenges when doing multimodal research.…”

mentioning

confidence: 99%

The quantification of gesture–speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking

2019

View full text Add to dashboard Cite

There is increasing evidence that hand gestures and speech synchronize their activity on multiple dimensions and timescales. For example, gesture's kinematic peaks (e.g., maximum speed) are coupled with prosodic markers in speech. Such coupling operates on very short timescales at the level of syllables (200 ms), and therefore requires high-resolution measurement of gesture kinematics and speech acoustics. High-resolution speech analysis is common for gesture studies, given that field's classic ties with (psycho)linguistics. However, the field has lagged behind in the objective study of gesture kinematics (e.g., as compared to research on instrumental action). Often kinematic peaks in gesture are measured by eye, where a "moment of maximum effort" is determined by several raters. In the present article, we provide a tutorial on more efficient methods to quantify the temporal properties of gesture kinematics, in which we focus on common challenges and possible solutions that come with the complexities of studying multimodal language. We further introduce and compare, using an actual gesture dataset (392 gesture events), the performance of two video-based motion-tracking methods (deep learning vs. pixel change) against a high-performance wired motion-tracking system (Polhemus Liberty). We show that the videography methods perform well in the temporal estimation of kinematic peaks, and thus provide a cheap alternative to expensive motion-tracking systems. We hope that the present article incites gesture researchers to embark on the widespread objective study of gesture kinematics and their relation to speech. Keywords Motion tracking. Video recording. Deep learning. Gesture and speech analysis. Multimodal language There is an increasing interest in the ways that cooccurring hand gestures and speech coordinate their activity (Esteve-Gibert & Guellaï, 2018; Wagner, Malisz, & Kopp, 2014). The nature of this coordination is exquisitely complex, as it operates on multiple levels and timescales. For example, on the semantic level, referential

show abstract

“…With computer vision (Cao, Simon, Wei, & Sheikh, 2017) we obtained motion traces of manual and head gestures (see e.g., Lepic, Börstell, Belsitzman, & Sandler, 2016;Ripperda, Drijvers, & Holler, 2020). We then investigated kinematic interrelationships between gestures (e.g., Beecks et al, 2015Beecks et al, , 2016Sato, Schouwstra, Flaherty, & Kirby, 2020), where we leverage bivariate time-series analysis (dynamic time warping (DTW)) with network topology analysis and visualization (Pouw & Dixon, 2019;Pouw et al, 2021). Through this analysis, we show that the study of gesture's form can reveal the linguistic constraints on the kinematic system as a whole.…”

Section: Current Studymentioning

confidence: 99%

A Systematic Investigation of Gesture Kinematics in Evolving Manual Languages in the Lab

et al. 2021

View full text Add to dashboard Cite

Silent gestures consist of complex multi-articulatory movements but are now primarily studied through categorical coding of the referential gesture content. The relation of categorical linguistic content with continuous kinematics is therefore poorly understood. Here, we reanalyzed the video data from a gestural evolution experiment (Motamedi, Schouwstra, Smith, Culbertson, & Kirby, 2019), which showed increases in the systematicity of gesture content over time. We applied computer vision techniques to quantify the kinematics of the original data. Our kinematic analyses demonstrated that gestures become more efficient and less complex in their kinematics over generations of learners. We further detect the systematicity of gesture form on the level of thegesture kinematic interrelations, which directly scales with the systematicity obtained on semantic coding of the gestures. Thus, from continuous kinematics alone, we can tap into linguistic aspects that were previously only approachable through categorical coding of meaning. Finally, going beyond issues of systematicity, we show how unique gesture kinematic dialects emerged over generations as isolated chains of participants gradually diverged over iterations from other chains. We, thereby, conclude that gestures can come to embody the linguistic system at the level of interrelationships between communicative tokens, which should calibrate our theories about form and linguistic content.

show abstract

Spatiotemporal Similarity Search in 3D Motion Capture Gesture Streams

Cited by 13 publications

References 42 publications

Content-based retrieval in videos from laparoscopic surgery

Content-based retrieval in videos from laparoscopic surgery

The quantification of gesture–speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking

A Systematic Investigation of Gesture Kinematics in Evolving Manual Languages in the Lab

Contact Info

Product

Resources

About