Applying Machine Learning to the Evaluation of Interviewer Performance

Abstract

Survey organizations have long used Computer Assisted Recorded Interviewing (CARI) to monitor interviewer performance. Conventionally, a human coder needs to first listen to the audio recording of the interactions between the interviewer and the respondent and then evaluate and code features of the question-and-answer sequence using a pre-specified coding scheme. Although prior research found that providing feedback to interviewers based on CARI was effective at improving interviewer performance, such coding process tends to be labor intensive and time consuming. To improve the effectiveness and efficiency of using CARI to monitor interviewer performance, we developed a pipeline that heavily draws on the use of machine learning to process audio recorded interviews. In particular, machine learning is used to detect who spoke at which turn in a question-level audio recording and to transcribe conversations at the turn level. This paper describes how the pipeline was used to detect interviewer falsification and to identify problematic interviewer behavior in both recordings of mock interviews and actual field interviews. The performance of the pipeline was discussed.

Applying Machine Learning to Survey Question Assessment

“…We enhanced the CARI ML pipeline described in Sun and Yan (2023) to include additional measures that can be used to detect questions with worse performance. We presented six measures that can be used to flag interactional difficulties and breakdowns in the question-answering process.…”

“…The analytics component of the pipeline is augmented to produce outcome metrics that can be used for either monitoring interviewer performance (Sun and Yan 2023) or assessing survey questions (Table 1).…”

“…We have provided technical details of our pipeline in this paper and Sun and Yan (2023). We encourage interested survey researchers to replicate our study and to use principles of the pipeline in their study for their particular goals of question evaluation and testing.…”

“…As shown in Figure 1, the expanded audio pipeline includes three component processes built on machine learning: speaker diarization, speech-to-text, and acoustic feature extraction. The first two components (speaker diarization and speech-to-text) are described in detail in Sun and Yan (2023). To recap, the speaker diarization process uses Pyannote.Audio pre-trained models (Bredin et al 2020) to determine who spoke at which turn in a questionanswer sequence.…”

“…The speech-to-text process uses Whisper pre-trained model (Radford et al 2023) to transcribe conversations at turn-level. In addition to the metrics described in Sun and Yan (2023), an extension built into the pipeline enables the pipeline to generate six new measures that can be used to assess interactional difficulties, as shown in Table 1.…”