Sequential pattern mining of multimodal data streams in dyadic interactions

Abstract

In this paper we propose a sequential pattern mining method to analyze multimodal data streams using a quantitative temporal approach. While the existing algorithms can only find sequential orders of temporal events, this paper presents a new temporal data mining method focusing on extracting exact timings and durations of sequential patterns extracted from multiple temporal event streams. We present our method with its application to the detection and extraction of human sequential behavioral patterns over multiple multimodal data streams in human-robot interactions. Experimental results confirmed the feasibility and quality of our proposed pattern mining algorithm, and suggested a quantitative data-driven way to ground social interactions in a manner that has never been achieved before.

What Leads To Shared Attention? Maternal Cues and Infant Responses During Object Play

“…We recorded infant–mother interactions at participants’ homes and coded participants’ actions frame‐by‐frame (30 Hz). Microbehavioral analyses can reveal input patterns that structure behaviors and promote learning (de Barbaro, Johnson, Forster, & Deák, ; Fricker, Zhang, & Yu, ; Hutchins, ; Kaye & Fogel, ). The current project uses microbehavioral analysis to document the prevalence, patterning, and effectiveness of the five actions described above, as cues that precede attention sharing.…”

Methodological Considerations For Investigating the Microdynamics of Social Interaction Development

“…Investigators should anticipate a laborious process of converging on a set of tools that will capture the temporal dynamics of interest. The importance of this process is recognized and has been articulated by others [39], [41], [92]. In the next section, we provide examples of this process from our own analyses.…”

A geometric approach for mining sequential patterns in interval-based data streams

“…Furthermore, an interval-based event can also be denoted as an event point on a 2-D plane with start time as the horizontal axis and end time as the vertical axis, known as Reich's plot [17]. ESMiner [18], [19] proposed to run clustering algorithm on the Reich's plot with all event points from the database and create frequent patterns in an Apriori manner. Fricker, Zhang and Yu [18] use a similar approach but first represent the intervals in a coordinate system based on their starting time and duration.…”

“…ESMiner [18], [19] proposed to run clustering algorithm on the Reich's plot with all event points from the database and create frequent patterns in an Apriori manner. Fricker, Zhang and Yu [18] use a similar approach but first represent the intervals in a coordinate system based on their starting time and duration. Then the results of geometric clustering will generate seed patterns for their APriori algorithm.…”