Singing Ability Assessment: Development and validation of a singing test based on item response theory and a general open-source software environment for singing data

Abstract

We describe the development of the Singing Ability Assessment (SAA) open-source test environment. The SAA captures and scores different aspects of human singing ability and melodic memory in the context of item response theory. Taking perspectives from both melodic recall and singing accuracy literature, we present results from two online experiments (N = 247; N = 910). On-the-fly audio transcription is produced via a probabilistic algorithm and scored via latent variable approaches. Measures of the ability to sing long notes indicate a three-dimensional principal components analysis solution representing pitch accuracy, pitch volatility and changes in pitch stability (proportion variance explained: 35%; 33%; 32%). For melody singing, a mixed-effects model uses features of melodic structure (e.g., tonality, melody length) to predict overall sung melodic recall performance via a composite score [R2c = .42; R2m = .16]. Additionally, two separate mixed-effects models were constructed to explain performance in singing back melodies in a rhythmic [R2c = .42; R2m = .13] and an arhythmic [R2c = .38; R2m = .11] condition. Results showed that the yielded SAA melodic scores are significantly associated with previously described measures of singing accuracy, the long note singing accuracy measures, demographic variables, and features of participants’ hardware setup. Consequently, we release five R packages which facilitate deploying melodic stimuli online and in laboratory contexts, constructing audio production tests, transcribing audio in the R environment, and deploying the test elements and their supporting models. These are published as open-source, easy to access, and flexible to adapt.

Learning and Recalling Melodies

“…This is at least the case for when melodies are long enough (e.g., 15-48 notes) to require multiple attempts to sing back in full. In other sung recall research with short unknown melodies 3-15 notes in length (Silas, Mu ¨llensiefen, & Kopiez, 2023;Silas, Robinson, et al, 2023), where it is conceivable to sing back a melody in one attempt, we have been able to successfully connect melodic features to melodic similarity (opti3) directly. This suggests that, particularly with longer melodies, general memory capacities are important to include in modeling, beyond musical features and musical memory (Silas et al, 2022).…”

“…The melodic feature predictors employed were taken from the FANTASTIC toolbox (Mu ¨llensiefen, 2009) (see Appendix listing #12 for online supplement source material). A priori, we chose i.entropy (to indicate the amount of ''surprise'' in intervallic information), d.entropy (to indicate the amount of ''surprise'' in rhythmic information), tonalness (to indicate the level of tonality), target melody length (to indicate overall constraint on working memory) and step.cont.loc.var (to indicate the amount of variation in contour) due to previous research indicating that they serve as good predictors of melodic memory (Dreyfus et al, 2016;Harrison et al, 2017;Mu ¨llensiefen & Halpern, 2014;Silas, Mu ¨llensiefen, & Kopiez, 2023). Additionally, to capture the reoccurrence of melodic patterns and the overall selfsimilarity of each melody (Deutsch, 1980), we compute the mean information content of each sequence of melodic pitches using the ppm R package (Harrison et al, 2020).…”

“…First, we suggest that attempt length and opti3 should be even more integratively modeled, using a much larger database of items and more heterogeneity in melodic features. We have recently implemented such a framework (Silas, Mu ¨llensiefen, & Kopiez, 2023). Second, a general working memory construct (measured by one or more variables) should be included as a predictor, as this may have some explanatory power aside from musical memory faculties (see Silas et al, 2022, for a discussion of these issues).…”