Assessing the Generalizability of Randomized Trial Results to Target Populations

Stuart, Elizabeth A.; Bradshaw, Catherine P.; Leaf, Philip J.

doi:10.1007/s11121-014-0513-z

“…The long-standing problem of generalizing experimental findings from the trial sample to the population as a whole, also known as the problem of "sample selection-bias" [1,2], has received renewed attention in the past decade, as more researchers come to recognize this bias as a major threat to the validity of experimental findings in both the health sciences [3] and social policy making [4]. Since participation in a randomized trial cannot be mandated, we cannot guarantee that the study population would be the same as the population of interest.…”

Section: Transportability and Selection Biasmentioning

confidence: 99%

“…For example, as more individuals become eligible for health insurance, the types of individuals seeking services would no longer match the type of individuals that were sampled for the study [3]. A similar change would occur as more individuals become aware of the efficacy of the treatment.…”

Section: Transportability and Selection Biasmentioning

confidence: 99%

Generalizing Experimental Findings

Pearl

¹

2015

Journal of Causal Inference

View full text Add to dashboard Cite

This note examines one of the most crucial questions in causal inference: "How generalizable are randomized clinical trials?" The question has received a formal treatment recently, using a non-parametric setting, and has led to a simple and general solution. I will describe this solution and several of its ramifications, and compare it to the way researchers have attempted to tackle the problem using the language of ignorability. We will see that ignorability-type assumptions need to be enriched with structural assumptions in order to capture the full spectrum of conditions that permit generalizations, and in order to judge their plausibility in specific applications.Keywords: generalizability, transportability, selection bias, admissibility, ignorability Transportability and selection biasThe long-standing problem of generalizing experimental findings from the trial sample to the population as a whole, also known as the problem of "sample selection-bias" [1,2], has received renewed attention in the past decade, as more researchers come to recognize this bias as a major threat to the validity of experimental findings in both the health sciences [3] and social policy making [4]. Since participation in a randomized trial cannot be mandated, we cannot guarantee that the study population would be the same as the population of interest. For example, the study population may consist of volunteers, who respond to financial and medical incentives offered by pharmaceutical firms or experimental teams, so, the distribution of outcomes in the study may differ substantially from the distribution of outcomes under the policy of interest.Another impediment to the validity of experimental finding is that the types of individuals in the target population may change over time [5]. For example, as more individuals become eligible for health insurance, the types of individuals seeking services would no longer match the type of individuals that were sampled for the study [3]. A similar change would occur as more individuals become aware of the efficacy of the treatment. The result is an inherent disparity between the target population and the population under study.The problem of generalizing across disparate populations has received a formal treatment in Pearl and Bareinboim [6] where it was labeled "transportability," and where necessary and sufficient conditions for valid generalization were established (see [7]). The problem of selection bias, though it has some unique features, can also be viewed as a nuance of the transportability problem, thus inheriting all the theoretical results established in Pearl and Bareinboim [6] that guarantee valid generalizations. I will describe the two problems side by side and then return to the distinction between the type of assumptions that are needed for enabling generalizations.The transportability problem concerns two dissimilar populations, Å and Å Ã , and requires us to estimate the average causal effect P Ã ðy x Þ (explicitly: The first two components of I TR represent, resp...

show abstract

“…The classical problem of generalizing experimental findings from the trial sample to the population as a whole, also known as the problem of "sample selection-bias" (Heckman, 1979;Bareinboim et al, 2014), has received renewed attention in the past decade, as more researchers come to recognize this bias as a major threat to the validity of experimental findings in both the health sciences (Stuart et al, 2015) and social policy making (Manski, 2013).…”

Section: Transportability and Selection Biasmentioning

confidence: 99%

“…For example, as more individuals become eligible for health insurance, the types of individuals seeking services would no longer match the type of individuals that were sampled for the study (Stuart et al, 2015). A similar change would occur as more individuals become aware of the efficacy of the treatment.…”

Section: Transportability and Selection Biasmentioning

confidence: 99%

See 1 more Smart Citation

Generalizing Experimental Findings

Pearl¹

2015

View full text Add to dashboard Cite

This note examines one of the most crucial questions in causal inference: "How generalizable are randomized clinical trials?" The question has received a formal treatment recently, using a non-parametric setting which has led to a simple and general solution. I will describe this solution and several of its ramifications, and compare it to the way researchers have attempted to tackle the problem using the language of ignorability. We will see that ignorability-type assumptions need to be enriched with structural assumptions in order to capture the full spectrum of conditions that permit generalizations, and in order to judge their plausibility in specific applications.

show abstract