Assessment of risk of bias is regarded as an essential component of a systematic review on the effects of an intervention. The most commonly used tool for randomised trials is the Cochrane risk-of-bias tool. We updated the tool to respond to developments in understanding how bias arises in randomised trials, and to address user feedback on and limitations of the original tool.

BackgroundThe National Institute for Health and Care Excellence (NICE) commissioned a ‘mock technology appraisal’ to assess whether changes to its methods and processes are needed. This report presents the findings of independent research commissioned to inform this appraisal and the deliberations of a panel convened by NICE to evaluate the mock appraisal.MethodsOur research included reviews to identify issues, analysis methods and conceptual differences and the relevance of alternative decision frameworks, alongside the development of an exemplar case study of chimeric antigen receptor (CAR) T-cell therapy for treating acute lymphoblastic leukaemia.ResultsAn assessment of previous evaluations of regenerative medicines found that, although there were a number of evidential challenges, none was unique to regenerative medicines or was beyond the scope of existing methods used to conceptualise decision uncertainty. Regarding the clinical evidence for regenerative medicines, the issues were those associated with a limited evidence base but were not unique to regenerative medicines: small non-randomised studies, high variation in response and the intervention subject to continuing development. The relative treatment effects generated from single-arm trials are likely to be optimistic unless it is certain that the historical data have accurately estimated the efficacy of the control agent. Pivotal trials may use surrogate end points, which, on average, overestimate treatment effects. To reduce overall uncertainty, multivariate meta-analysis of all available data should be considered. Incorporating indirectly relevant but more reliable (more mature) data into the analysis can also be considered; such data may become available as a result of the evolving regulatory pathways being developed by the European Medicines Agency. For the exemplar case of CAR T-cell therapy, target product profiles (TPPs) were developed, which considered the ‘curative’ and ‘bridging to stem-cell transplantation’ treatment approaches separately. Within each TPP, three ‘hypothetical’ evidence sets (minimum, intermediate and mature) were generated to simulate the impact of alternative levels of precision and maturity in the clinical evidence. Subsequent assessments of cost-effectiveness were undertaken, employing the existing NICE reference case alongside additional analyses suggested within alternative frameworks. The additional exploratory analyses were undertaken to demonstrate how assessments of cost-effectiveness and uncertainty could be impacted by alternative managed entry agreements (MEAs), including price discounts, performance-related schemes and technology leasing. The panel deliberated on the range of TPPs, evidence sets and MEAs, commenting on the likely recommendations for each scenario. The panel discussed the challenges associated with the exemplar and regenerative medicines more broadly, focusing on the need for a robust quantification of the level of uncertainty in the cost-effective estimates and the potential value of MEAs in limiting the exposure of the NHS to high upfront costs and loss associated with a wrong decision.ConclusionsIt is to be expected that there will be a significant level of uncertainty in determining the clinical effectiveness of regenerative medicines and their long-term costs and benefits, but the existing methods available to estimate the implications of this uncertainty are sufficient. The use of risk sharing and MEAs between the NHS and manufacturers of regenerative medicines should be investigated further.FundingThe National Institute for Health Research Health Technology Assessment programme.

SummaryObjectiveTo compare the effectiveness of acupuncture with other relevant physical treatments for alleviating pain due to knee osteoarthritis.DesignSystematic review with network meta-analysis, to allow comparison of treatments within a coherent framework. Comprehensive searches were undertaken up to January 2013 to identify randomised controlled trials in patients with osteoarthritis of the knee, which reported pain.ResultsOf 156 eligible studies, 114 trials (covering 22 treatments and 9,709 patients) provided data suitable for analysis. Most trials studied short-term effects and many were classed as being of poor quality with high risk of bias, commonly associated with lack of blinding (which was sometimes impossible to achieve). End of treatment results showed that eight interventions: interferential therapy, acupuncture, TENS, pulsed electrical stimulation, balneotherapy, aerobic exercise, sham acupuncture, and muscle-strengthening exercise produced a statistically significant reduction in pain when compared with standard care. In a sensitivity analysis of satisfactory and good quality studies, most studies were of acupuncture (11 trials) or muscle-strengthening exercise (9 trials); both interventions were statistically significantly better than standard care, with acupuncture being statistically significantly better than muscle-strengthening exercise (standardised mean difference: 0.49, 95% credible interval 0.00–0.98).ConclusionsAs a summary of the current available research, the network meta-analysis results indicate that acupuncture can be considered as one of the more effective physical treatments for alleviating osteoarthritis knee pain in the short-term. However, much of the evidence in this area of research is of poor quality, meaning there is uncertainty about the efficacy of many physical treatments.