Context
Schizophrenia is a mental disorder which affects the way a person thinks, feels and acts. It is characterised by abnormal social behaviour and may lead to difficulties in distinguishing between what is real and what is imaginary. Schizophrenia has been ranked among the top causes of disability in the world (Murray et al., 1996, Tandon et al., 2008).
There are a wide range of competing antipsychotic drugs available in the market, and there have been many randomised controlled trials (RCTs) that assess most of the available treatment options. The RCTs cover a wide range of treatment comparisons, forming a network of evidence (see Overview of Evidence Synthesis and NMA for a description of network meta-analysis). In addition, there have been non-randomised studies (NRSs) measuring the effectiveness of drugs in real-world clinical settings. However, the two different types of evidence have not been jointly synthesised. The benefits of adding NRS, a type of real-world data (RWD), to the synthesis is explained on NMA incorporating RWE. For more information on combining randomised and non-randomised evidence in NMA, see Efthimiou et al., 2017.
What was examined in this case study?
The aim of this case study was to assess existing methodology and develop new methods for combining evidence from RCTs and NRSs in a network meta-analysis (NMA). Specific issues examined were:
- How can inconsistencies between the different types of evidence (randomised and non-randomised) be assessed?
- What analytic methods can be used to incorporate RWE from NRSs into an NMA?
- How to choose between the methods?
- How to quantify the influence of the observational evidence in the pooled NMA results?
The data used came from both aggregate data from RCTs and both aggregate and individual-participant data (IPD) from published non-randomised studies. Three generic approaches for incorporating RWD in a NMA of RCTs were explored:
- design-adjusted approach: data from real-world studies are ‘shifted’ and down-weighted based on external opinion about their credibility. This approach was recommended when resources allow for a separate assessment of bias for each non-randomised study
- use of informative priors: in this approach non-randomised evidence was used to construct informative prior distributions for the basic parameters
- three-level hierarchical models: these models are more appropriate when data from studies of several different designs are to be synthesised and account for heterogeneity within and across designs.
What ‘effectiveness challenge(s)’ was addressed in this case study?
Practical applications of NMA are usually limited to synthesising evidence from RCTs. NMAs frequently disregard observational evidence from NRSs because it is commonly assumed that estimates of relative treatment effects are more likely to be biased, especially when confounding has been inadequately addressed.
When non-randomised evidence is included in an NMA, this increases concerns about intransitivity and inconsistency of the method, and that results may be very precise, yet biased. However, interest in including NRSs in the NMA synthesis and decision-making process is growing.
Although RCTs are the most reliable source of information on relative treatment effects, their strictly experimental setting and inclusion criteria may limit their ability to predict results in real-world clinical practice. NRS-based estimates of treatment effects may complement evidence provided by RCTs, and potentially address some of their limitations.
What were the findings and conclusions?
The advantages and limitations of each method are discussed in detail, and extensive guidance is provided on the most appropriate method to use in each scenario. The results of the case study indicated that the inclusion of a well-conducted observational study can corroborate findings of an NMA based on RCTs alone, increase precision and enhance the decision-making process.
Key points that researchers may want to consider when planning to combine randomised and non-randomised evidence in a NMA:
- Adjusting estimates from NRSs should always take place when possible to minimize the risk of bias. Availability of individual participant data is needed for this. Adjusted estimates may still be biased due to residual confounding. The extent and directionality of this bias may be hard to assess.
- Randomised and non-randomised evidence should first be analysed separately and results should be scanned for important discrepancies.
- If no inconsistencies are found across the different sources of evidence for each treatment comparison, a synthesis of all available evidence in a NMA can be performed using an array of different approaches.
- Network meta-regression can be used to account for differences in the populations of patients included in the RCTs vs. real-world studies.
- All approaches explored allow for a range of sensitivity analyses to control the impact of the non-randomised evidence in the pooled estimates in relative effects. Such sensitivity analyses are necessary to assess the impact of possible biases in the observational evidence.
- A choice between the various approaches should be primarily dictated by ease of interpretation and the resources available in the review team. Choices should be clearly described in the protocol.
What do stakeholders say?
For stakeholder feedback on the methods used in this case study, see NMA incorporating RWE.
Key contributor
Orestis Efthimiou, University of Bern and University of Ioannina School of Medicine