The Toybox Dataset of Egocentric Visual Object Transformations

Wang, Xiaohan; Ma, Tengyu; Ainooson, James; Cha, Seunghwan; Wang, Xiaotian; Molla, Azhar; Kunda, Maithilee

doi:10.48550/arxiv.1806.06034

Cited by 3 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…ToyBox. Motivated by recreating the naturalistic patterns of embodied visual experience, Wang et al [2018] created an egocentric video dataset called Toybox that contains egocentric (i.e., first-person perspective) videos of common household objects and toys being manually manipulated to undergo structured transformations, such as rotation, translation, and zooming.…”

Section: Related Work: Related Benchmarksmentioning

confidence: 99%

ModelVsBaby: a Developmentally Motivated Benchmark of Out-of-Distribution Object Recognition

Sheybani,

Maini,

Dendukuri

et al. 2024

Preprint

View full text Add to dashboard Cite

Deep neural networks have recently become remarkable computational tools for thinking about human visual learning. Recent studies have explored the effects of altering naturalistic images and compared the responses of both humans and models, providing valuable insights into their functioning and how deep neural networks can shape our understanding of human learning. Critically, much of human visual learning happens throughout early development. Yet, well-controlled benchmarks comparing AI models with young humans are scarce. Here, we present a developmentally motivated benchmark of out-of-distribution (OOD) object recognition. Our benchmark, ModelVsBaby, includes a set of OOD conditions that have long been studied in the vision science literature, and are expected to be sensitive to the development of OOD object-recognition in humans: silhouette, geon, occluded, blurred, crowded background, and a baseline realistic condition. Along with the stimuli, we release a unique dataset of the responses of 2-year-old children to the stimuli. Our preliminary analyses of the dataset show several interesting patterns: 2-year-olds achieve 80% accuracy in the silhouette condition, nearly as well as in the realistic condition (chance=12%). They also perform well above chance, near 60% accuracy, on the other challenging conditions. We also evaluate image-text association (CLIP) models trained on varying amounts of internet-scale datasets. The model performances show that with enough data, all conditions are learnable by artificial learners. However, Realistic and Silhouette are learned with fewer training data similar to humans. Our benchmark stimuli and infant responses, provide an essential steppingstone for building computational models that are aligned with humans both in terms of the learning outcomes as well as the learning trajectory. This endeavor can furnish creating better models of visual development as well as improving the efficiency of AI systems for practical applications. Future work may use the benchmark stimuli to test more age groups, and provide a detailed comparison of models of various flavors in terms of “developmental alignment".

show abstract

Section: Related Work: Related Benchmarksmentioning

confidence: 99%

ModelVsBaby: a Developmentally Motivated Benchmark of Out-of-Distribution Object Recognition

Sheybani,

Maini,

Dendukuri

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…We follow the work [11] to come up with the ordering, training and testing data splits. The Toybox dataset [34] contains videos of toy objects from 12 classes. We used a subset of the full dataset containing 348 toy objects with 10 instances per object, each containing a spatial transformation of that object's pose.…”

Section: Datasetsmentioning

confidence: 99%

“…We evaluated our method under two typical stream learning protocols (Figure 1), incremental class iid and incremental class instance, across three benchmark datasets, CORe50 [21], Toybox [34] and iLab [3]. The performance of HAMN is comparable to or even better than state-of-the-art (SOTA) methods.…”

Section: Introductionmentioning

confidence: 99%

Hypothesis-driven Online Video Stream Learning with Augmented Memory

Zhang¹,

Badkundri²,

Talbot³

et al. 2021

Preprint

View full text Add to dashboard Cite

“…To mimic patterns of visual experience infants get when playing with objects, the Toybox dataset (Wang et al 2018) was curated in our lab. The dataset contains egocentric videos of several toy objects from a small number of categories being manipulated in different ways.…”

Section: Introductionmentioning

confidence: 99%

Learning from an Infant’s Visual Experience

Sanyal

2024

AAAI

View full text Add to dashboard Cite

Infants see a selective view of the world: they see some objects with high frequency and from a wide range of viewpoints (e.g., their toys during playing) while a much larger set of objects are seen much more rarely and from limited viewpoints (e.g., objects they see outdoors). Extensive, repeated visual experiences with a small number of objects during infancy plays a big role in the development of human visual skills. Internet-style datasets that are commonly used in computer vision research do not contain the regularities that result from such repeated, structured experiences with a few objects. This has led to a dearth of models that learn by exploiting these regularities. In my PhD dissertation, I use deep learning models to investigate how regularities in an infant's visual experience can be leveraged for visual representation learning.

show abstract

The Toybox Dataset of Egocentric Visual Object Transformations

Cited by 3 publications

References 15 publications

ModelVsBaby: a Developmentally Motivated Benchmark of Out-of-Distribution Object Recognition

ModelVsBaby: a Developmentally Motivated Benchmark of Out-of-Distribution Object Recognition

Hypothesis-driven Online Video Stream Learning with Augmented Memory

Learning from an Infant’s Visual Experience

Contact Info

Product

Resources

About