What is it?
Evidence synthesis is the process of retrieving, evaluating and summarising the findings of all relevant studies on a certain subject area.
When the primary aim of evidence synthesis is to estimate the effect between two interventions, this can be achieved by a systematic review of relevant randomised controlled trials (RCTs) and synthesis of the RCT results using meta-analytical techniques (in a pairwise meta-analysis). A weighted summary can then be calculated by accounting for the precision of each retrieved treatment effect and for the potential for between-study heterogeneity.
However, when there are multiple treatments available for the same disease, network meta-analysis (NMA), an extension of the usual meta-analysis, may be used (see below for a description of NMA). Although NMA is often applied to summarise aggregate data (for example, published estimates of relative treatment effects), it is also possible to combine the raw data from each trial by undertaking an individual participant data (IPD) meta-analysis. IPD can improve the analyses and may provide more reliable results. NMA using IPD are especially useful for estimating relative treatment effects from non-randomised studies of interventions (NRSs) other sources of real-world data (RWD) to help account for the lack of randomisation and to adjust for possible confounding.
Applications of meta-analysis and NMA are mostly limited to the synthesis of evidence from RCTs. However, there is a growing interest in the medical community for incorporating evidence from NRSs, patient registries and other RWD sources. This strategy is particularly appealing when there are few RCTs to answer a specific research question. Non-randomised evidence may also complement randomised evidence when the available RCTs do not align with the target population, prescription strategies or primary outcomes of the research question.
Indirect treatment comparison and network meta-analysis
Meta-analysis is a widely accepted statistical tool, used for synthesising evidence on the relative effects of interventions obtained from multiple individual RCTs. However, the value of pairwise meta-analysis may be limited in real-world clinical practice, if there are multiple competing interventions for a condition and the available RCTs do not include some of the pairwise comparisons. In such cases, it is possible to synthesise all of the available evidence from RCTs by undertaking an NMA.
NMA uses two types of evidence; direct and indirect. A treatment comparison, B vs. C, may be carried out directly, by studies comparing the two treatments head-to-head. This comparison may also be carried out indirectly, by comparing treatments B and C with a third treatment, A (B vs. A and C vs. A) – an indirect treatment comparison. In the diagram below (figure 1), for B vs. C there is both direct evidence (solid line) and indirect evidence via treatment A (dashed line). These two sources of evidence can be combined into a mixed treatment comparison.
Figure 1. A direct ‘head-to-head’ comparison (right) vs an indirect treatment comparison (left)
In practice, there might be many competing interventions for a condition, and the available RCTs may not cover all of the pairwise comparisons. This is shown in the diagram below, in which there is a network of six treatments (A–F) and a set of studies comparing different pairs of treatments (connecting lines). The studies include some, but not all of the pairwise comparisons (for example, there are no RCTs directly comparing C vs. B, C vs. D or E vs. B). For any treatment comparison there may be direct and multiple indirect estimates. In such complicated cases of data availability, an NMA can be used to synthesise all of the evidence and provide a ranking of all available treatments. NMA offers many advantages, resulting in a rapid growth of the number published.
Figure 2. Network of six treatments (bold lines indicate RCTs comparing treatments)
For more information on best practice for indirect treatment comparisons and NMA, please see here. The GetReal review on NMA methods can be found here and all articles identified in this review can be found here.
Why is it useful?
- Summarises the evidence: the available evidence on the effects of an intervention can be summarised.
- Assesses reproducibility and generalisability: the reproducibility and generalisability of individual study findings can be assessed.
- Identifies heterogeneity in treatment effects: sources of heterogeneity in treatment effects can be identified.
NMA in particular:
- Increased precision and power: NMAs can have greater precision and power compared with a series of pairwise meta-analyses. This is achieved by synthesising both direct and indirect evidence on treatment comparisons in a single analysis.
- Allows indirect comparison: NMA can be used to compare interventions that have not been compared directly in head-to-head trials.
- Ranks treatments: a ranking of all competing treatments can be provided by NMA.
- Reduces controversy: NMA can address controversies between individual studies.
- Avoids selective use of data: by including all of the available evidence, NMA can help to avoid the selective use of data in decision-making.
- Combines all of the evidence: all of the available evidence is synthesised together in a joint analysis.
When is it useful?
- Conflicting evidence: if there is conflicting evidence about the relative effect of treatments, meta-analysis may help to address this.
- Direct comparisons are not available: when there are multiple competing treatments for a disease and the available RCTs compare them to placebo or standard care for regulatory purposes, but not to each other, NMA can be especially useful.
- Evidence only from comparisons with older or less effective treatments: when the best available treatments for a disease are not compared with each other, and only comparisons with older or less effective treatments are available, NMA methods can provide clinically relevant estimates to better support decision-making.
- Bias in direct comparisons: under certain circumstances, the use of indirect comparisons in NMA may help counterbalance biases in the direct pairwise comparisons.
What are the limitations?
- Not equivalent to direct evidence from RCTs: indirect comparisons and the resulting estimates are, by nature, observational and should not be considered to be as robust as results from RCTs comparing interventions directly with each other (in head-to-head trials).
- Transitivity is assumed: NMA methods rely on the assumption of transitivity, i.e. that B vs. C can be compared via A. Transitivity can be hard to assess and justify, and this is sometimes considered to be an important limitation. If the transitivity assumption is not justified, NMA results may be biased.
- Difficulties in interpretation: when inconsistencies are present in the network (i.e. direct and indirect estimates are in disagreement), NMA results are hard to interpret and may be considered unreliable.
- Complex to carry out: performing an NMA needs greater statistical expertise and is more complicated than performing a series of pairwise meta-analyses.
- Low return for effort: results for the relative treatment effects of the comparisons may only marginally change, despite the additional efforts needed to perform a NMA compared with pairwise meta-analysis.
What do stakeholders say?
- Some stakeholders still remain sceptical on the validity of indirect comparisons and NMA.
- For some stakeholders NMA methods remain a ‘black box’ and they are reluctant to accept NMA results.
What techniques for evidence synthesis are available?
The specific technique or analytical method used for the synthesis of evidence will depend on the nature of the data available.
Please see the table below for more information on the different techniques which could be used for evidence synthesis, depending on the form and source of the data available. In addition, see here for a decision tree on conducting network-meta-analysis using RWD.
Table. Evidence synthesis techniques by form and source of data available.
|Form of data|
|Aggregate||IPD||Aggregate + IPD|
Source of data
|RCT only||See references here.||See description here.
See GetReal case study here.
|See references here.|
|Real-world data (with or without RCT)||See references here.||See description here.||See description here.|
Thomas Debray, University Medical Center Utrecht
Orestis Efthimiou, University of Bern and University of Ioannina School of Medicine