Evaluating Bang for the Buck

Abstract

Evaluators often use qualitative research methods, yet there is little evidence on the comparative cost-effectiveness of the two most commonly employed qualitative methods-in-depth interviews (IDIs) and focus groups (FGs). We performed an inductive thematic analysis of data from 40 IDIs and 40 FGs on the health-seeking behaviors of African American men (N ¼ 350) in Durham, North Carolina. We used a bootstrap simulation to generate 10,000 random samples from each data set and calculated the number of data collection events necessary to reach different levels of thematic saturation. The median number of data collection events required to reach 80% and 90% saturation was 8 and 16, respectively, for IDIs and 3 and 5 for FGs. Interviews took longer but were more costeffective at both levels. At the median, IDIs cost 20-36% less to reach thematic saturation. Evaluators can consider these empirically based cost-effectiveness data when selecting a qualitative data collection method.

A simple method to assess and report thematic saturation in qualitative research

“…Similarly, Guest et al [9] conducted a stepwise inductive thematic analysis of 60 in-depth interviews among female sex workers in West Africa and discovered that 70% of all 114 identified themes turned up in the first six interviews, and 92% were identified within the first 12 interviews. Subsequent studies by Francis et al and Namey et al [17,18] reported similar findings. Building on these earlier studies, Hagaman and Wutich [19] calculated saturation within a cross-cultural study and found that fewer than 16 interviews were enough to reach data saturation at each of the four sites but that 20-40 interviews were necessary to identify cross-cultural meta-themes across sites.…”

Using the cognitive interviewing process to improve survey design by allied health: A qualitative study

“…While only conducting six interviews, this method gave the researchers some ability to see if the amendments they had made had resolved the identified issues. Future research could investigate the optimal number of rounds of cognitive interviewing that are necessary to ensure saturation is reached (Guest, Bunce & Johnson, ) while also conducting a cost effectiveness analysis based on the thematic saturation level of each interview round (Namey, Guest, McKenna & Chen, ). These investigations would identify if the cost of research methodology outweighs the benefits from the survey improvements.…”

Physicians’ perspectives regarding pragmatic clinical trials

“…Codes were iteratively revised and refined by the study team over the course of coding the initial interviews. Each transcript was independently coded by at least two team members who met to compare codes and resolve disagreements [17,18]. All transcripts were then entered into NVivo 10 software (QSR International Pty Ltd, Cambridge, MA, USA).…”