Quantifying Thematic Saturation in Qualitative Data Analysis

Abstract

An important aspect of qualitative research is reaching saturation-loosely, a point at which observing more data will not lead to discovery of more information related to the research questions. However, there has been no validated means of objectively establishing saturation. This article proposes a novel quantitative approach to measuring thematic saturation based on a sound statistical model. The model is validated on two data sets from different qualitative research projects involving interviews, focus groups, and literature surveys. The proposed model provides consistent estimates of saturation across data sets, within branches of data sets, and over the course of a research project. The model can be used for both quantifying saturation and estimating the number of observations required to achieve a specified level of saturation.

Open-ended interview questions and saturation

“…Similarly, Lowe et al [ 25 ] estimated saturation and domain size in two examples and in simulated datasets, testing the effect of various parameters. Lowe et al found that responses were not independent across respondents and that saturation may never be reached.…”

When I say … quantification in qualitative research

“…There has even been a recent call for the measuring of saturation through a so-called "saturation index" as a "validated means of objectively establishing saturation". 3 The problem is that fuzzy boundaries exist between words and numbers: qualitative and quantitative data are neither static nor binary. 2 So, although there is a multiplicity of ways in which qualitative research is quantified, and the use of numbers in qualitative research may cause us to gasp, what do we mean when we say … quantification in qualitative research?…”

“…However, when working within an epistemic framework underpinned by subjectivity and multiple realities, quantification can actually serve to undermine more complex and nuanced interpretations of the meaning of qualitative data, thereby undermining the quality of our qualitative work. 3 For example, misrepresentation of qualitative data can occur when numbers (such as frequencies and percentages of themes) are employed in contexts in which participants have not been asked the same questions. 1,4 So, given the benefits but also the pitfalls of using numbers in qual- percentages should be employed only in the context of larger qualitative samples (ie n > 50), and (iv) that semi-quantification (the use of words like "majority", "minority", "few", "some" and "many") should be used only with justification.…”

“…Other, relatively uncommon, uses of quantification in qualitative research include the establishing of inter‐rater reliability in the context of coding by multiple analysts. There has even been a recent call for the measuring of saturation through a so‐called “saturation index” as a “validated means of objectively establishing saturation” . The problem is that fuzzy boundaries exist between words and numbers: qualitative and quantitative data are neither static nor binary .…”

“…Here, we are often fully aware of the situation, but we choose to play the numbers game for practical reasons. However, when working within an epistemic framework underpinned by subjectivity and multiple realities, quantification can actually serve to undermine more complex and nuanced interpretations of the meaning of qualitative data, thereby undermining the quality of our qualitative work . For example, misrepresentation of qualitative data can occur when numbers (such as frequencies and percentages of themes) are employed in contexts in which participants have not been asked the same questions …”