Network meta-analysis using individual participant data from RCTs

What is it?

Network meta-analysis (NMA) is a common approach to synthesise the efficacy of multiple treatments, and to compare their relative efficacy (see description of NMA here). NMA can be based on the raw individual participant data (IPD) from each study, but most are based on aggregate data (that is, published summary estimates of relative treatment effect, usually as identified through a systematic literature search). The GetReal review on methods for IPD meta-analysis which includes a section on multiple treatment comparisons can be found here and the articles identified in this review can be found here.

Why is it useful?

  • Increased precision and consistency: NMA based on IPD may give more precise estimated results and greater consistency than NMA based on aggregate data because participant-level characteristics can be better adjusted for. NMA based on aggregate data tends to have higher levels of inconsistency and/or between-study heterogeneity, imprecise treatment effects (meaning that there is low confidence in the results) and may yield biased results, giving invalid or misleading estimates of comparative treatment effects. See the box below.
  • Supports inclusion of real-world evidence (RWE): Access to IPD may support the inclusion of RWE in analyses. More information on the use of NMA using RWE can be found here.
  • Increases ability to do the following:
    • analyse all important outcomes
    • determine the validity of analysis assumptions
    • use the same unit and method of analysis for each trial
    • conduct more detailed analysis of time-to-event outcomes
    • achieve greater power for assessing interactions between effects of interventions and participant or disease/condition characteristics (known as effect modification)
    • conduct more complex analyses
    • use non-standard models or measures of effect. (Tierney et al 2015, Debray et al 2015)
  • May help personalise treatment strategies: NMA based on IPD may help to tailor treatment strategies by:
    • including evidence from studies with diverse populations (for example, non-randomised studies of interventions) (Debray et al 2015)
    • adjusting for factors that may influence the outcome (known as prognostic factors)
    • adjusting for treatment-covariate interactions
    • borrowing information across studies (for example, when outcomes of primary interest have not been fully measured in randomised controlled trials [RCTs]). (Debray et al 2016 and GetReal case study)
  • May also help to:
    • clarify trial eligibility with trial investigators
    • improve the quality of the data (for example, to include trials that are not reported in full).
    • standardise outcome definitions across trials or translate different definitions to a common scale
    • update follow-up or time-to-event outcomes beyond those reported
    • clarify the trial design, conduct and analysis methods
    • overcome potential bias arising from drop-out related to the treatment (known as informative drop-out). (Tierney et al 2015, Debray et al 2015)

Box: Issues with NMA based on aggregate data

NMA based on aggregate data tends to have:

  • Higher levels of inconsistency: this means that the comparative treatment effect of two interventions differ according to whether they are assessed head-to-head or indirectly through other treatment comparisons (see diagrams of networks here).
  • High between-study heterogeneity: this implies that the comparative treatment effect of two interventions substantially differs across the included RCTS.
  • Very imprecise treatment effects: meaning that there is very low confidence in the results from NMA based on aggregate data.
  • Biased results: meaning that estimates of comparative treatment effect are invalid or misleading.

(note: The presence of inconsistency and heterogeneity in an NMA may degrade the validity and clinical usefulness of summary estimates of comparative treatment effect, and should therefore be properly addressed.)

When is it useful?

  • When NMA based on aggregate data are affected by inconsistency and/or between-study differences in estimates of comparative treatment effect.
  • When NMA based on aggregate data yield imprecise treatment effects.
  • When the analysis methods of individual trials are deemed inappropriate.
  • When NMA are based on non-randomised studies of interventions.
  • To investigate the presence of treatment effect modification.
  • When a new medicine is being developed and tested in randomised trials, NMA based on IPD may help to assess its potential added value and to identify subgroups where its efficacy appears most promising.

What are the limitations?

  • More time and effort are needed when carrying out NMA based on IPD to make agreements with investigators, organise data collection, clean data and perform analyses.
  • Obtaining IPD for all relevant studies is usually not feasible.
  • Studies without accessible IPD may differ in results from those that have accessible IPD (creating bias).
  • Analysis may not be straightforward when using sources that report their data differently.
  • NMA based on IPD cannot compensate for poorly designed and conducted primary research.

Key contributor

Thomas Debray, University Medical Center Utrecht