What is it?
Evidence synthesis is the process of retrieving, evaluating, combining and summarising the findings of all relevant studies on a certain subject area.
When the primary aim of evidence synthesis is to estimate the relative effect of two interventions, this can be achieved by a systematic review of relevant randomised controlled trials (RCTs) and synthesis of the RCT results using meta-analytical techniques, conventionally in a pairwise meta-analysis. A summary value for the effect can then be calculated, with associated ranges of uncertainty to account for the precision of each treatment effect reported in the studies included in the meta-analysis, and weightings may be applied to account for potential for between-study heterogeneity.
However, when there are multiple treatments available for the same disease, indirect comparisons and network meta-analysis (NMA), an extension of conventional pairwise meta-analysis, may be used. See below for a description of NMA. Although NMA is often used to summarise aggregate data, for example, published estimates of relative treatment effects, it is also possible to combine raw data from each trial by undertaking an individual participant data (IPD) meta-analysis. Use of IPD can improve the precision of analyses and may provide more reliable results.
NMA using IPD are especially useful for estimating relative treatment effects from non-randomised studies of interventions (NRSs) to help account for the lack of randomisation and to incorporate adjustments for possible confounding.
Applications of meta-analysis and NMA are mostly limited to the synthesis of evidence from RCTs. However, when there are none or few RCTs to answer a specific research question, there is a growing interest in developing NMAs to incorporate evidence from NRSs, patient registries and other RWD sources. Non-randomised evidence may also be used in NMAs to complement randomised evidence when available RCTs fail to align with the research question in terms of target population, prescription strategies or definition of (primary) outcomes.
Indirect treatment comparison and network meta-analysis
Meta-analysis is a widely accepted statistical tool, used for synthesising evidence on the relative effects of interventions obtained from multiple individual RCTs. However, the value of pairwise meta-analysis may be limited in real-world clinical practice, if there are multiple competing interventions for a condition and the available RCTs do not include some of the pairwise comparisons. In such cases, it is possible to synthesise all of the available evidence from RCTs by undertaking an NMA.
NMAs combines two types of evidence: direct and indirect. A treatment comparison, B vs. C, may be carried out directly, by studies comparing the two treatments head-to-head. This comparison may also be carried out indirectly, by making use of comparisons of each treatments, B and C, with a third treatment, A (B vs. A and C vs. A) – an indirect treatment comparison. In the diagram below (figure 1), for B vs. C there is both direct evidence (solid line) and indirect evidence via treatment A (dashed line). These two types of evidence can be combined into a mixed treatment comparison (of which NMA is one type).
Figure 1. A direct ‘head-to-head’ comparison (right) vs an indirect treatment comparison (left)
In practice, there might be many competing interventions for a condition, and the available RCTs may not cover all of the pairwise comparisons. This is shown in figure 2 below, in which there is a network of six treatments (A–F) and a set of studies directly comparing different pairs of treatments (connecting lines). The studies include some, but not all of the pairwise comparisons (for example, there are no RCTs directly comparing C vs. B, C vs. D or E vs. B). For any treatment comparison there may be direct estimates and a variety of indirect estimates. In such more complicated cases of data availability, an NMA can be used to synthesise all of the evidence and provide a ranking of all available treatments. NMA has proved to be a very useful method for synthesising evidence, resulting in a rapid growth of the number of studies published in recent years.
Figure 2. Network of six treatments (bold lines indicate RCTs comparing treatments)
For more information on best practice for indirect treatment comparisons and NMA, see Best Practice in Indirect Comparisons.
The GetReal review on NMA methods can be found in Efthimiou et al, 2016, and all articles identified in this review can be found on the Zotero Library for WP4 Network Meta-Analysis.
Why is it useful?
Meta-analysis:
- Summarises the evidence: the available evidence on the effects of an intervention can be summarised.
- Assesses reproducibility and generalisability: the reproducibility and generalisability of individual study findings can be assessed.
- Identifies heterogeneity in treatment effects: the presence of heterogeneity in treatment effects can be identified, and sources can be explored.
NMA in particular:
- Increased precision and power: NMAs can have greater precision and power compared with a series of pairwise meta-analyses. This is achieved by synthesising both direct and indirect evidence on treatment comparisons in a single analysis.
- Allows indirect comparison: NMA can be used to compare interventions that have not been compared directly in head-to-head trials.
- Ranks treatments: a ranking of all competing treatments with respect the outcome of interest can be provided by NMA.
- Reduces controversy: NMA can help to resolve controversies between proponents of individual studies (with conflicting results).
- Avoids selective use of data: by including all of the available evidence, NMA can help to avoid the selective use of data in decision-making.
- Combines all of the evidence: all of the available evidence is synthesised together in a joint analysis.
When is it useful?
- Conflicting evidence: if there is conflicting evidence about the relative effect of treatments, meta-analysis may help to address this.
- Direct comparisons are not available: when there are multiple competing treatments for a disease and the available RCTs compare them to placebo or standard care for regulatory purposes, but not to each other, NMA can be especially useful.
- Evidence only from comparisons with older or less effective treatments: when the best available treatments for a disease are not compared with each other, and only comparisons with older or less effective treatments are available, NMA can provide clinically relevant estimates to better support decision-making.
- Bias in direct comparisons: under certain circumstances, the use of indirect comparisons in NMA may help counterbalance biases in the direct pairwise comparisons.
What are the limitations?
- Not equivalent to direct evidence from RCTs: indirect comparisons and the resulting estimates are, by nature, observational and should not be considered to be as robust as results from RCTs comparing interventions directly with each other (in head-to-head trials).
- Transitivity is assumed: NMA methods rely on the assumption of transitivity, i.e. that there are no differences in the distribution of effect modifiers for trials of A vs. B and A vs. C, so that these trials can indeed be used to estimate B vs. C. Transitivity can be hard to assess and justify, requiring clinical as well as statistical judgments. If the transitivity assumption is not justified, NMA results may be biased.
- Difficulties in interpretation: when inconsistencies are present in the network (i.e. direct and indirect estimates are in disagreement), NMA results are hard to interpret and may be considered unreliable.
- Complex to carry out: performing an NMA needs greater statistical expertise and is more complicated than performing a series of pairwise meta-analyses.
- Low return for effort: results for the relative treatment effects of the comparisons may only marginally change, despite the additional efforts needed to perform an NMA compared with pairwise meta-analysis.
What do stakeholders say?
- Some stakeholders still remain sceptical about the validity of indirect comparisons and NMA.
- For some stakeholders NMA methods remain a ‘black box’ and they are reluctant to accept NMA results.
What techniques for evidence synthesis are available?
The specific technique or analytical method used for the synthesis of evidence will depend on the nature of the data available.
Please see the table below for more information on the different techniques which could be used for evidence synthesis, depending on the form and source of the data available. In addition, see the Decision Tree on Conducting Network Meta-Analysis using RWD.
Table. Evidence synthesis techniques by form and source of data available.
| Form of data | ||||
| Aggregate | IPD | Aggregate + IPD | ||
|
Source of data |
RCT only | See References for Evidence Synthesis Techniques not covered by GetReal | For a description, see NMA using IPD from RCTs
For a case study, see Case Study: Methods for NMA using IPD – Depression |
See References for Evidence Synthesis Techniques not covered by GetReal |
| Real-world data (with or without RCT) | See References for Evidence Synthesis Techniques not covered by GetReal | For a description, see NMA incorporating RWE
For examples of GetReal case studies, see Case Study: Incorporating Non-Randomised Studies in NMA of RCTs – Schizophrenia and Case Study: Using RWE to Connect Networks of Evidence – Rheumatoid Arthritis |
For a description, see NMA incorporating RWE
For examples of GetReal case studies, see Case Study: Incorporating Non-Randomised Studies in NMA of RCTs – Schizophrenia and Using RWE to Connect Networks of Evidence – Rheumatoid Arthritis |
|
Key contributors
Thomas Debray, University Medical Center Utrecht
Orestis Efthimiou, University of Bern and University of Ioannina School of Medicine