Developmental psychologists should adopt citizen science to improve generalization and reproducibility

Li, Wei; Germine, Laura; Mehr, Samuel A.; Srinivasan, Mahesh; Hartshorne, Joshua K.

doi:10.1002/icd.2348

Cited by 13 publications

(8 citation statements)

References 199 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our approach instead relies on an unmoderated, citizen-science approach. While this approach has the advantage of larger-scale, and often more diverse recruitment than in-person or moderated online studies (Hilton & Mehr, 2022; Li et al, 2022), it has two potential risks.…”

Section: Methodsmentioning

confidence: 99%

Children infer the behavioral contexts of unfamiliar foreign songs.

Hilton¹,

Thierry²,

Yan³

et al. 2023

Journal of Experimental Psychology: General

Self Cite

View full text Add to dashboard Cite

Music commonly appears in behavioral contexts in which it can be seen as playing a functional role, as when a parent sings a lullaby with the goal of soothing a baby. Humans readily make inferences, based on the sounds they hear, regarding the behavioral contexts associated with music. These inferences tend to be accurate, even if the songs are in foreign languages or unfamiliar musical idioms; upon hearing a Blackfoot lullaby, a Korean listener with no experience of Blackfoot music, language, or broader culture is far more likely to judge the music's function as "used to soothe a baby" than "used for dancing". Are such inferences shaped by musical exposure or does the human mind naturally detect links between musical form and function of these kinds? Children's developing experience of music provides a clear test of this question. We studied musical inferences in a large sample of children recruited online (N = 5,033), who heard dance, lullaby, and healing songs from 70 world cultures and who were tasked with guessing the original behavioral context in which each was performed. Children reliably inferred the original behavioral contexts with only minimal improvement in performance from the youngest (age 4) to the oldest (age 16), providing little evidence for an effect of experience. Children's inferences tightly correlated with those of adults for the same songs, as collected from a similar online experiment (N = 98,150). Moreover, similar acoustical features were predictive of the inferences of both samples. These findings suggest that accurate inferences about the behavioral contexts of music, driven by universal links between form and function in music across cultures, do not always require extensive musical experience.

show abstract

Section: Methodsmentioning

confidence: 99%

Children infer the behavioral contexts of unfamiliar foreign songs.

Hilton¹,

Thierry²,

Yan³

et al. 2023

Journal of Experimental Psychology: General

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the first 18 months after deployment, that team gathered more than 40 million moral judgments from over 4 million unique participants in 233 countries and territories (Awad, Dsouza, Bonnefon, Shariff, & Rahwan, 2020). Recruiting such large sample sizes from volunteers is appealing; however, success with such recruitment requires participant-reward strategies like gamification or personalized feedback (Hartshorne et al, 2019; Li, Germine, Mehr, Srinivasan, & Hartshorne, 2022). Thus, it has been hard to generalize the model to other important research questions and experiments, particularly when taking part in the experiment does not appear to be fun or interesting.…”

Section: Critiques and Concernsmentioning

confidence: 99%

Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences

et al. 2022

View full text Add to dashboard Cite

The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as “playing twenty questions with nature,” theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. The researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm—and with far greater efficiency.

show abstract

“…This latter technique, which we will refer to as manual eyetracking, has previously been applied in lab settings (e.g., Trueswell, 2008;Snedeker & Trueswell, 2004;Yacovone et al, 2021; sometimes called poor-man's eye-tracking). The benefits of web-based research methods using off-the-shelf hardware are obvious: by programming an experiment that participants can complete from wherever they are, researchers can recruit a much larger and more diverse sample than is feasible in lab-based studies (e.g., Hartshorne et al, 2018;Li et al, 2022). In addition, there is no need for expensive hardware, and these techniques involve equipment that is easily portable (a laptop with a webcam), thereby additionally allowing researchers to recruit participants in settings such as museums, schools, or field sites.…”

Section: Introductionmentioning

confidence: 99%

Webcams as windows to the mind? A direct comparison between in-lab and web-based eye-tracking methods

Slim,

Kandel,

Yacovone

et al. 2024

Preprint

View full text Add to dashboard Cite

There is a growing interest in the use of webcams to conduct eye-tracking experiments over the internet. We assessed the performance of two webcam-based eye-tracking techniques for behavioral research: manual annotation of webcam videos (manual eye-tracking) and the automated WebGazer eye-tracking algorithm. We compared these methods to a traditional infrared eye-tracker and assessed their performance in both lab and web-based settings. In both lab and web experiments, participants completed the same battery of five tasks, selected to trigger effects of various sizes: two visual fixation tasks and three visual world tasks testing real-time (psycholinguistic) processing effects. In the lab experiment, we simultaneously collected infrared eye-tracking, manual eye-tracking, and WebGazer data; in the web experiment, we simultaneously collected manual eye-tracking and WebGazer data. We found that the two webcam-based methods are suited to capture different types of eye-movement patterns. Manual eye-tracking, similar to infrared eye-tracking, detected both large and small effects. WebGazer, however, showed less accuracy in detecting short, subtle effects. There was no notable effect of setting for either method. We discuss the trade-offs researchers face when choosing eye-tracking methods and offer advice for conducting eye-tracking experiments over the internet.

show abstract

Developmental psychologists should adopt citizen science to improve generalization and reproducibility

Cited by 13 publications

References 199 publications

Children infer the behavioral contexts of unfamiliar foreign songs.

Children infer the behavioral contexts of unfamiliar foreign songs.

Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences

Webcams as windows to the mind? A direct comparison between in-lab and web-based eye-tracking methods

Contact Info

Product

Resources

About