A thousand studies for the price of one: Accelerating psychological science with Pushkin

Abstract

Half of the world's population has internet access. In principle, researchers are no longer limited to subjects they can recruit into the laboratory. Any study that can be run on a computer or mobile device can be run with nearly any demographic anywhere in the world, and in large numbers. This has allowed scientists to effectively run hundreds of experiments at once. Despite their transformative power, such studies remain rare for practical reasons: the need for sophisticated software, the difficulty of recruiting so many subjects, and a lack of research paradigms that make effective use of their large amounts of data, due to such realities as that they require sophisticated software in order to run effectively. We present Pushkin: an open-source platform for designing and conducting massive experiments over the internet. Pushkin allows for a wide range of behavioral paradigms, through integration with the intuitive and flexible jsPsych experiment engine. It also addresses the basic technical challenges associated with massive, worldwide studies, including auto-scaling, extensibility, machine-assisted experimental design, multisession studies, and data security. Keywords Online studies. Robust and reliable research. Massive online experiments. Citizen science Although some questions psychologists care about involve comparing only two conditions to each other, most require teasing apart the contributions of many intertwined variables. In the past, this has required hundreds, if not thousands, of studies across numerous laboratories, each targeting a specific variable, population, or stimulus set. In principle, we can now do this work many orders of magnitude more quickly. Given that half the world's population has internet access (ITU Telecommunication Development Sector, 2017), any study that can be run on a computer or mobile device can be run with nearly any demographic anywhere in the world, and in large numbers. This includes not just surveys, but studies involving grammatical judgments, reaction times, decisionmaking, economics games, eyetracking, priming, sentence completion, skill acquisition, and others-which is to say, most human behavioral experiments (Birnbaum, 2004;

Acoustic regularities in infant-directed speech and song across cultures

“…We analyzed all data available at the time of writing this manuscript from the “Who’s Listening?” game at https://themusiclab.org/quizzes/ids, a jsPsych 67 experiment distributed via Pushkin 68 to both desktop and mobile web browsers. Participants ( N = 13,218; gender: 4,405 female, 7,043 male, 176 other, 1,594 did not disclose; age: median 31 years, interquartile range 23-43) listened to at least 1 and at most 16 vocalizations drawn at random from the corpus, for a total of 164,759 ratings (infant-directed song: n = 47,798; infant-directed speech: n = 38,913; adult-directed song: n = 41,277; adult-directed speech: n = 37,071).…”

Universality and diversity in human song

“…We analyzed all data available at the time of writing this manuscript from participants in the "World Music Quiz", hosted on the citizen science website http://themusiclab.org. The site runs on the Pushkin platform (139), which presents experiments in desktop or mobile web browsers, playing audio and recording participant responses using the jsPsych library (140) In contrast to Experiment 1 of previous work with the NHS Discography (54), where listeners rated each excerpt on 6 different dimensions (i.e., they rated how much they thought the song could be used to soothe a baby, for dancing, and so on), listeners were asked to guess which of the four song types they had just heard. Participants could only provide one response per song.…”

Universality and diversity in human song

“…We analyzed all data available at the time of writing this manuscript from participants in the "World Music Quiz", hosted on the citizen science website http://themusiclab.org. The site runs on the Pushkin platform (139), which presents experiments in desktop or mobile web browsers, playing audio and recording participant responses using the jsPsych library (140). Participants (N = 29,357; 8,203 female, 15,946 male, 341 other, 4,867 did not disclose; median age 33 years, interquartile range: 25-45, 1st percentile: 12, 99th percentile: 74) listened to at least 1 song and at most 8 songs (per session) drawn at random from the NHS Discography (median 8 plays, interquartile range: 5-8).…”