Assessing observational studies of medical treatments

Hartz, Arthur J.; Bentler, Suzanne E.; Charlton, Mary E.; Lanska, Douglas J.; Butani, Yogita; Soomro, G M; Benson, Kjell

doi:10.1186/1742-7622-2-8

Cited by 28 publications

(10 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[30][31][32][33][34][35][36][37][38][39][40][41][42][43] A summary of the methods and findings of each the 14 identified studies is presented in Appendix 2 (see Table 42). …”

Section: Quantification Of Bias In Observational Studiesmentioning

confidence: 99%

See 1 more Smart Citation

The assessment and appraisal of regenerative medicines and cell therapy products: an exploration of methods for review, economic evaluation and appraisal

Hettle

Corbett

Hinde

et al. 2017

Health Technol Assess

106

143

View full text Add to dashboard Cite

BackgroundThe National Institute for Health and Care Excellence (NICE) commissioned a ‘mock technology appraisal’ to assess whether changes to its methods and processes are needed. This report presents the findings of independent research commissioned to inform this appraisal and the deliberations of a panel convened by NICE to evaluate the mock appraisal.MethodsOur research included reviews to identify issues, analysis methods and conceptual differences and the relevance of alternative decision frameworks, alongside the development of an exemplar case study of chimeric antigen receptor (CAR) T-cell therapy for treating acute lymphoblastic leukaemia.ResultsAn assessment of previous evaluations of regenerative medicines found that, although there were a number of evidential challenges, none was unique to regenerative medicines or was beyond the scope of existing methods used to conceptualise decision uncertainty. Regarding the clinical evidence for regenerative medicines, the issues were those associated with a limited evidence base but were not unique to regenerative medicines: small non-randomised studies, high variation in response and the intervention subject to continuing development. The relative treatment effects generated from single-arm trials are likely to be optimistic unless it is certain that the historical data have accurately estimated the efficacy of the control agent. Pivotal trials may use surrogate end points, which, on average, overestimate treatment effects. To reduce overall uncertainty, multivariate meta-analysis of all available data should be considered. Incorporating indirectly relevant but more reliable (more mature) data into the analysis can also be considered; such data may become available as a result of the evolving regulatory pathways being developed by the European Medicines Agency. For the exemplar case of CAR T-cell therapy, target product profiles (TPPs) were developed, which considered the ‘curative’ and ‘bridging to stem-cell transplantation’ treatment approaches separately. Within each TPP, three ‘hypothetical’ evidence sets (minimum, intermediate and mature) were generated to simulate the impact of alternative levels of precision and maturity in the clinical evidence. Subsequent assessments of cost-effectiveness were undertaken, employing the existing NICE reference case alongside additional analyses suggested within alternative frameworks. The additional exploratory analyses were undertaken to demonstrate how assessments of cost-effectiveness and uncertainty could be impacted by alternative managed entry agreements (MEAs), including price discounts, performance-related schemes and technology leasing. The panel deliberated on the range of TPPs, evidence sets and MEAs, commenting on the likely recommendations for each scenario. The panel discussed the challenges associated with the exemplar and regenerative medicines more broadly, focusing on the need for a robust quantification of the level of uncertainty in the cost-effective estimates and the potential value of MEAs in limiting the exposure of the NHS to high upfront costs and loss associated with a wrong decision.ConclusionsIt is to be expected that there will be a significant level of uncertainty in determining the clinical effectiveness of regenerative medicines and their long-term costs and benefits, but the existing methods available to estimate the implications of this uncertainty are sufficient. The use of risk sharing and MEAs between the NHS and manufacturers of regenerative medicines should be investigated further.FundingThe National Institute for Health Research Health Technology Assessment programme.

show abstract

“…[30][31][32][33][34][35][36][37][38][39][40][41][42][43] A summary of the methods and findings of each the 14 identified studies is presented in Appendix 2 (see Table 42). …”

Section: Quantification Of Bias In Observational Studiesmentioning

confidence: 99%

“…In six of the studies, [33][34][35]37,38,41 data were sourced from published meta-analyses that included both RCTs and NRSs. Five other studies 30,32,39,40,42 took a different approach and searched for NRSs that compared treatment effects and then carried out a further search to locate relevant RCTs.…”

Section: Quantification Of Bias In Observational Studiesmentioning

confidence: 99%

The assessment and appraisal of regenerative medicines and cell therapy products: an exploration of methods for review, economic evaluation and appraisal

Hettle

Corbett

Hinde

et al. 2017

Health Technol Assess

106

143

View full text Add to dashboard Cite

show abstract

“…The reliance on RCTs as the highest level of evidence is therefore being challenged [27]; even regulatory agencies are now beginning to review evidence from well-designed observational research when making labeling evaluations. On the contrary, the quality of design and reporting of many observational studies has been questioned [28]. The same attention to design, the control of confounding factors, and complete reporting are clearly as necessary for observational studies as for RCTs.…”

Section: Limitations Of Current Guidelinesmentioning

confidence: 99%

Breaking new ground: challenging existing asthma guidelines

Price

Thomas

2006

BMC Pulm Med

View full text Add to dashboard Cite

BackgroundWhile we have international guidelines and various national guidelines for asthma diagnosis and management, asthma remains poorly controlled in many children and adults. In this paper we review the limitations of current asthma guidelines and describe important issues and remaining questions regarding asthma guidelines for use, particularly in primary care.DiscussionClinical practice guidelines based on evidence from randomized controlled trials are considered the most rigorous and accurate. Current evidence-based guidelines are written predominantly from the perspective of the patient with a clear-cut asthma diagnosis, however, and tend not to consider the heterogeneity of asthma or to accommodate individual patient variations in response to treatment or their needs, differences in practice settings, or local differences in availability and cost of therapies. The results of randomized controlled trials, which are designed to establish efficacy of treatment under ideal conditions, may not apply to 'real-world' clinical practice, where patients are unselected, monitoring is less frequent, and effectiveness – the benefit of treatment in routine clinical practice – is the most relevant outcome. Moreover, most guidelines see asthma in isolation rather than considering other factors that may impact on asthma and response to asthma therapy, particularly age, allergic rhinitis, cigarette smoking, adherence, and genetic factors. When these links are recognized, guidelines rarely provide practical recommendations for treatment in these scenarios. Finally, there is some evidence that general practitioners are not convinced of the applicability of asthma guidelines to their practice settings, especially when those writing the guidelines principally work in specialist practice.ConclusionDeveloping country-specific guidelines or, ideally, local guidelines could provide more practical solutions for asthma care and could account for regional factors that influence patient choice and adherence to therapy. Pragmatic clinical trials and well-designed observational trials are needed in addition to randomized controlled trials to assess real-world effectiveness of therapies, and such evidence needs also to be considered by guideline writers. Finally, practical tools to facilitate the diagnosis and assessment of asthma and factors responsible for poor control, such as associated allergic rhinitis, limited adherence, and smoking behavior, are needed to supplement treatment information provided in clinical practice guidelines for asthma.

show abstract

“…However, the concept that the assignment of subjects randomly to either experimental or control groups is a perfect science also has been questioned. In contrast to Hartz et al's assessment in 2005 75, Benson and Hartz 76 in a 2000 publication comparing observational studies and randomized controlled trials found little evidence that estimates of treatment effects in observational studies reported after 1984, were either consistently larger than or qualitatively different from those obtained in randomized controlled trials. Furthermore, Hartz et al 77, in a 2003 publication assessing observational studies of chemonucleolysis, concluded that the results suggested that a review of several comparable observational studies may help evaluate treatment, identify patient types most likely to benefit from a given treatment, and provide information about study features that can improve the design of subsequent observational studies or even randomized controlled trials.…”

Section: Discussionmentioning

confidence: 78%

“…The poor quality of reporting in observational intervention studies was reported as a potential factor for confounding bias in 98% of studies 74. In a 2005 publication, Hartz et al 75 assessed observational studies of medical treatments and concluded that reporting was often inadequate for use in comparing the study designs or allowing for any other meaningful interpretation of the results. However, the concept that the assignment of subjects randomly to either experimental or control groups is a perfect science also has been questioned.…”

Section: Discussionmentioning

confidence: 99%

Assessment of Effectiveness of Percutaneous Adhesiolysis in Managing Chronic Low Back Pain Secondary to Lumbar Central Spinal Canal Stenosis

Manchikanti

Cash²,

McManus³

et al. 2013

Int. J. Med. Sci.

View full text Add to dashboard Cite

Background: Chronic persistent low back and lower extremity pain secondary to central spinal stenosis is common and disabling. Lumbar surgical interventions with decompression or fusion are most commonly performed to manage severe spinal stenosis. However, epidural injections are also frequently performed in managing central spinal stenosis. After failure of epidural steroid injections, the next sequential step is percutaneous adhesiolysis and hypertonic saline neurolysis with a targeted delivery. The literature on the effectiveness of percutaneous adhesiolysis in managing central spinal stenosis after failure of epidural injections has not been widely studied.Study Design: A prospective evaluation.Setting: An interventional pain management practice, a specialty referral center, a private practice setting in the United States.Objective: To evaluate the effectiveness of percutaneous epidural adhesiolysis in patients with chronic low back and lower extremity pain with lumbar central spinal stenosis.Methods: Seventy patients were recruited. The initial phase of the study was randomized, double-blind with a comparison of percutaneous adhesiolysis with caudal epidural injections. The 25 patients from the adhesiolysis group continued with follow-up, along with 45 additional patients, leading to a total of 70 patients. All patients received percutaneous adhesiolysis and appropriate placement of the Racz catheter, followed by an injection of 5 mL of 2% preservative-free lidocaine with subsequent monitoring in the recovery room. In the recovery room, each patient also received 6 mL of 10% hypertonic sodium chloride solution, and 6 mg of non-particulate betamethasone, followed by an injection of 1 mL of sodium chloride solution and removal of the catheter.Outcomes Assessment: Multiple outcome measures were utilized including the Numeric Rating Scale (NRS), the Oswestry Disability Index 2.0 (ODI), employment status, and opioid intake with assessment at 3, 6, and 12, 18 and 24 months post treatment. The primary outcome measure was 50% or more improvement in pain scores and ODI scores.Results: Overall, a primary outcome or significant pain relief and functional status improvement of 50% or more was seen in 71% of patients at the end of 2 years. The overall number of procedures over a period of 2 years were 5.7 ± 2.73.Limitations: The lack of a control group and a prospective design.Conclusions: Significant relief and functional status improvement as seen in 71% of the 70 patients with percutaneous adhesiolysis utilizing local anesthetic steroids and hypertonic sodium chloride solution may be an effective management strategy in patients with chronic function limiting low back and lower extremity pain with central spinal stenosis after failure of conservatie management and fluoroscopically directed epidural injections.

show abstract

Assessing observational studies of medical treatments

Cited by 28 publications

References 64 publications

The assessment and appraisal of regenerative medicines and cell therapy products: an exploration of methods for review, economic evaluation and appraisal

The assessment and appraisal of regenerative medicines and cell therapy products: an exploration of methods for review, economic evaluation and appraisal

Breaking new ground: challenging existing asthma guidelines

Assessment of Effectiveness of Percutaneous Adhesiolysis in Managing Chronic Low Back Pain Secondary to Lumbar Central Spinal Canal Stenosis

Contact Info

Product

Resources

About