Full Description of An Automated Pipeline for Providing Personalized Feedback Based on Audio Samples

Abstract

Personalized feedback based on the automated analysis of au- dio samples could be useful in a wide range of intervention contexts, from early childhood to neurodegenerative programs, which target behaviors having vocal correlates. In this paper, we describe an automated pipeline that allows one to provide personalized feedback based on the automated analysis of au- dio samples of caregiver-child conversations captured using a smartphone. The pipeline relies on open-source packages and AWS in order to provide a cheap, reproducible, and consider- ably scalable solution for researchers and practitioners inter- ested in early childhood development and caregiver-child in- teraction, and which could be adapted for other use cases. It processes conversation files that are 1-10 minutes long, with a cost of 0.20 US$ per hour of audio analyzed. It is currently op- erational in one large-scale experiment in Uruguay, where audio files are collected through a chatbot, whose implementation is not covered in this paper. Finally, we lay out limitations of our approach and potential improvements.