The Human Speechome Project

Roy, Deb; Patel, Rupal; DeCamp, Philip; Kubat, Rony; Fleischman, Michael; Roy, Brandon; Mavridis, Nikolaos; Tellex, Stefanie; Salata, Alexia; Guinness, Jethran; Levit, Michael; Gorniak, Peter

doi:10.1007/11880172_15

Cited by 112 publications

(57 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Video was collected from eleven ceiling-mounted cameras and fourteen microphones were installed in a home as part of an effort to understand a child's language acquisition [12]. Sample frames from the corpus are shown in Figures 1 and 9.…”

Section: Task Constrained Corpusmentioning

confidence: 99%

Grounding spatial language for video search

Tellex

Kollar²,

Shaw

et al. 2010

International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction

Self Cite

View full text Add to dashboard Cite

The ability to find a video clip that matches a natural language description of an event would enable intuitive search of large databases of surveillance video. We present a mechanism for connecting a spatial language query to a video clip corresponding to the query. The system can retrieve video clips matching millions of potential queries that describe complex events in video such as "people walking from the hallway door, around the island, to the kitchen sink." By breaking down the query into a sequence of independent structured clauses and modeling the meaning of each component of the structure separately, we are able to improve on previous approaches to video retrieval by finding clips that match much longer and more complex queries using a rich set of spatial relations such as "down"and "past." We present a rigorous analysis of the system's performance, based on a large corpus of task-constrained language collected from fourteen subjects. Using this corpus, we show that the system effectively retrieves clips that match natural language descriptions: 58.3% were ranked in the top two of ten in a retrieval task. Furthermore, we show that spatial relations play an important role in the system's performance. Categories and Subject Descriptors General Terms algorithms, experimentation, measurementPermission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ICMI-MLMI'10, November 8-12, 2010, Beijing, China. Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$10.00. Highest-ranked clipLow-ranked clips Figure 1: Results from the system for the query "from the couches in the living room to the dining room table." The person's start location is marked with a green dot; the end location is marked with a red dot, and their trajectory is marked in white.

show abstract

Section: Task Constrained Corpusmentioning

confidence: 99%

Grounding spatial language for video search

Tellex

Kollar²,

Shaw

et al. 2010

International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction

Self Cite

View full text Add to dashboard Cite

show abstract

“…This scenario has received considerable recent attention, in part because media collections have outgrown the amount of reliable annotation. Such videos are available on the web [14], in benchmark activities [139], and from individual research projects such as the Human SpeechHome [115]. There is a large variety of problems and solutions in this area, although it is relatively new.…”

Section: Unsupervised Event Discoverymentioning

confidence: 99%

Event Mining in Multimedia Streams

2008

View full text Add to dashboard Cite

| Events are real-world occurrences that unfold over space and time. Event mining from multimedia streams improves the access and reuse of large media collections, and it has been an active area of research with notable recent progress. This paper contains a survey on the problems and solutions in event mining, approached from three aspects: event description, event-modeling components, and current event mining systems. We present a general characterization of multimedia events, motivated by the maxim of five BW[s and one BH[ for reporting real-world events in journalism: when, where, who, what, why, and how. We discuss the causes for semantic variability in real-world descriptions, including multilevel event semantics, implicit semantics facets, and the influence of context. We discuss five main aspects of an event detection system. These aspects are: the variants of tasks and event definitions that constrain system design, the media capture setup that collectively define the available data and necessary domain assumptions, the feature extraction step that converts the captured data into perceptually significant numeric or symbolic forms, statistical models that map the feature representations to richer semantic descriptions, and applications that use event metadata to help in different information-seeking tasks. We review current event-mining systems in detail, grouping them by the problem formulations and approaches. The review includes detection of events and actions in one or more continuous sequences, events in edited video streams, unsupervised event discovery, events in a collection of media objects, and a discussion on ongoing benchmark activities. These problems span a wide range of multimedia domains such as surveillance, meetings, broadcast news, sports, documentary, and films, as well as personal and online media collections. We conclude this survey with a brief outlook on open research directions.

show abstract

“…To that end, we developed the Speechome Recorder (SR) (Figure 1), a portable version of the Speechome audio/video recording technology developed for the Human Speechome Project (HSP) [22]. Launched in 2005, the goal of the HSP was to study early language development through collection and analysis of longitudinal, ultra-dense and naturalistic recordings daily in the home of a family with a young child.…”

Section: Introductionmentioning

confidence: 99%

A portable audio/video recorder for longitudinal study of child development

Vosoughi

Goodwin

Washabaugh

et al. 2012

Proceedings of the 14th ACM International Conference on Multimodal Interaction

Self Cite

View full text Add to dashboard Cite

Collection and analysis of ultra-dense, longitudinal observational data of child behavior in natural, ecologically valid, non-laboratory settings holds significant promise for advancing the understanding of child development and developmental disorders such as autism. To this end, we created the Speechome Recorder -a portable version of the embedded audio/video recording technology originally developed for the Human Speechome Project -to facilitate swift, costeffective deployment in home environments. Recording child behavior daily in these settings will enable detailed study of developmental trajectories in children from infancy through early childhood, as well as typical and atypical dynamics of communication and social interaction as they evolve over time. Its portability makes possible potentially large-scale comparative study of developmental milestones in both neurotypical and developmentally delayed children. In brief, the Speechome Recorder was designed to reduce cost, complexity, invasiveness and privacy issues associated with naturalistic, longitudinal recordings of child development.

show abstract

The Human Speechome Project

Cited by 112 publications

References 4 publications

Grounding spatial language for video search

Grounding spatial language for video search

Event Mining in Multimedia Streams

A portable audio/video recorder for longitudinal study of child development

Contact Info

Product

Resources

About