Methodological Aspects in the Assessment of Treatment Effects in Observational Health Outcomes Studies

Abstract

Prospective observational studies, which provide information on the effectiveness of interventions in natural settings, may complement results from randomised clinical trials in the evaluation of health technologies. However, observational studies are subject to a number of potential methodological weaknesses, mainly selection and observer bias. This paper reviews and applies various methods to control for selection bias in the estimation of treatment effects and proposes novel ways to assess the presence of observer bias. We also address the issues of estimation and inference in a multilevel setting. We describe and compare the use of regression methods, propensity score matching, fixed-effects models incorporating investigator characteristics, and a multilevel, hierarchical model using Bayesian estimation techniques in the control of selection bias. We also propose to assess the existence of observer bias in observational studies by comparing patient- and investigator-reported outcomes. To illustrate these methods, we have used data from the SOHO (Schizophrenia Outpatient Health Outcomes) study, a large, prospective, observational study of health outcomes associated with the treatment of schizophrenia. The methods used to adjust for differences between treatment groups that could cause selection bias yielded comparable results, reinforcing the validity of the findings. Also, the assessment of observer bias did not show that it existed in the SOHO study. Observational studies, when properly conducted and when using adequate statistical methods, can provide valid information on the evaluation of health technologies.

Symptomatic remission in previously untreated patients with schizophrenia: 2-year results from the SOHO study

“…However, as reported previously (Haro et al , 2006b, the differences between treatment groups in the physician-rated CGI have also been seen in patient-rated measures, such as the health-related quality of life EQ-5D VAS. This is a powerful indicator that observed bias did not influence the results.…”

The Challenge of Selection Bias and Confounding in Palliative Care Research

“…The objective is to include a set of variables that are theoretically or actually correlated with both the intervention and the outcome to reduce the bias of the estimate of the treatment effect. 23,24 Bias refers to the difference between the estimated mean value and the "true" value (which can never actually be known). Including more potential confounders in the regression may decrease the bias of the treatment effect; however, adding more variables can decrease statistical power in small samples because it increases the variance (spread) around the regression estimate by decreasing the number of degrees of freedom.…”

“…This method allows investigators to examine and control for the distribution of confounders in both intervention and comparison groups by providing a summary measure of the conditional probability of being assigned to the intervention group (regardless of actual group assignment) based on a set of confounders. [23][24][25][26][27][28][29] The scores range from 0 to 1; the score for a particular participant represents the estimated probability of being assigned to the intervention group, given that person's particular combination of covariates. Participants with the same set of covariates will have the same score.…”

Effects of Enhanced Foster Care on the Long-term Physical and Mental Health of Foster Care Alumni

“…Although propensity score weighting was used to adjust for preplacement differences, this adjustment corrected for measured preplacement variables only. 37 Such adjustment is often no better in improving the accuracy of the estimate than the more conventional adjustment obtained using multivariate controls in conventional regression analyses. 38 It is possible that other unmeasured preplacement differences between the 2 samples could have introduced bias into the comparisons.…”