Conducting spoken word recognition research online: Validation and a new timing method

Abstract

Models of spoken word recognition typically make predictions that are then tested in the laboratory against the word recognition scores of human subjects (e.g., Luce & Pisoni Ear and Hearing, 19, 1-36, 1998). Unfortunately, laboratory collection of large sets of word recognition data can be costly and time-consuming. Due to the numerous advantages of online research in speed, cost, and participant diversity, some labs have begun to explore the use of online platforms such as Amazon's Mechanical Turk (AMT) to source participation and collect data (Buhrmester, Kwang, & Gosling Perspectives on Psychological Science, 6, 3-5, 2011). Many classic findings in cognitive psychology have been successfully replicated online, including the Stroop effect, task-switching costs, and Simon and flanker interference (Crump, McDonnell, & Gureckis PLoS ONE, 8, e57410, 2013). However, tasks requiring auditory stimulus delivery have not typically made use of AMT. In the present study, we evaluated the use of AMT for collecting spoken word identification and auditory lexical decision data. Although online users were faster and less accurate than participants in the lab, the results revealed strong correlations between the online and laboratory measures for both word identification accuracy and lexical decision speed. In addition, the scores obtained in the lab and online were equivalently correlated with factors that have been well established to predict word recognition, including word frequency and phonological neighborhood density. We also present and analyze a method for precise auditory reaction timing that is novel to behavioral research. Taken together, these findings suggest that AMT can be a viable alternative to the traditional laboratory setting as a source of participation for some spoken word recognition research.

Auditory presentation and synchronization in Adobe Flash and HTML5/JavaScript Web experiments

“…Third, researchers have legitimate concerns about the uncontrolled environment in which Web-based studies are run (for a more detailed overview, see Slote & Strand, 2016). Although the appearance of visual stimuli varies substantially from system to system, in terms of size, hue, saturation, contrast, among other things, the fact that people need to be able to view a display in order to use a computer means that the basic properties of a stimulus will be perceived by a participant.…”

“…However in the intervening 20 years, very little published research appears to have presented auditory stimuli over the Web (for overviews, see Knoll, Uther, & Costall, 2011, and Slote & Strand, 2016), and less still has examined the accuracy of doing so, systematically.…”

“…Auditory stimuli have included excerpts of musical performance (Honing, 2006), unpleasant sounds such as vomiting and dentists’ drills (Cox, 2008), and speech (Knoll et al, 2011; Slote & Strand, 2016). …”

“…More recently, Slote and Strand (2016) examined whether it would be possible to measure RTs to auditory stimuli online. In their Experiment 1, participants were presented with auditory consonant-vowel-consonant words such as “fit.” In the identification condition, participants had to identify and type in the word presented with a background babble added.…”

Support for context effects on segmentation and segments depends on the context

“…The studies that do exist have generally compared Mechanical Turk participants to participants collected before the advent of Mechanical Turk (Kleinschmidt & Jaeger, 2012) or have not used a comparison group at all (Kurumada, Brown, & Tanenhaus, 2012). However, the idea that Mechanical Turk participants may be useful for experiments in phonetics has recently gained some support from a careful validation of Mechanical Turk of previous language-related experiments using word identification and lexical decision tasks (Slote & Strand, 2016). Since there was an opportunity to use participants from Internet-based and live groups in this study, they were used to explore whether participants recruited through Mechanical Turk would show the same effects that were found in the lab.…”