What is it?
Network meta-analyses (NMAs) are often limited to synthesising evidence from randomised controlled trials (RCTs) (see Overview of evidence synthesis and network meta-analysis). However, interest in including real-world data (RWD) sources in NMA synthesis and decision-making is growing.
RCTs are considered the most reliable source of information on relative treatment effects, but their strictly experimental setting and inclusion criteria may limit their ability to predict results in real-world clinical practice. NMAs frequently exclude evidence from non-randomised studies (NRSs) or other sources of RWD because estimates of relative treatment effects are considered more likely to be biased, especially when confounding has not been adequately addressed. However, estimates of treatment effects from these other sources may be used to complement evidence provided by RCTs, and potentially address some of their limitations.
Why is it useful?
- More direct and relevant results: RCTs often have strict inclusion criteria, which may lead to the study populations differing from real-world populations. NRSs or other RWD sources may provide a more direct answer to the research question, and including RWE in the network may allow researchers to obtain more relevant answers.
- Increased precision and power: the inclusion of NRSs or other RWD sources can increase precision and power as compared with NMAs of RCTs.
- Potentially corroborative to RCT evidence alone: an NMA that includes both randomised and non-randomised evidence may corroborate conclusions drawn from an NMA of RCTs alone and reassure decision makers that study findings are transferable to real-world populations.
When is it useful?
- Lack of RCTs: including NRSs or other RWD sources in an NMA can be valuable when there are few RCTs available in the evidence-base.
- RCT network is disconnected: including NRSs or other RWD sources may be useful when the network of RCTs has two or more disconnected parts and the RWD source provides information on the missing links in the network. This allows the comparison of treatments that could not be compared using RCTs alone, and it may lead to increased the precision of the estimates of relative treatment effects. This was examined in the GetReal case study – Using RWE to connect networks of evidence – rheumatoid arthritis.
- Evidence is lacking: including NRSs or other RWD sources can be of value when the evidence from RCTs is sparse, for example, when investigating rare events.
- Research on harms of treatment: research questions on the harms of interventions are considered to be less at risk of selection bias in observational studies. In such cases NRSs or other RWD sources may constitute important sources of information.
- RCTs in limited populations: when RCTs are limited to highly selected populations, NRSs or other RWD sources may be more relevant to the research question and can provide information on the interventions in a real-world clinical setting.
What are its limitations?
- Risk of bias: estimates of relative treatment effects obtained from NRSs or other RWD sources are considered to be at a higher risk of bias, due to the lack of randomisation and increased risk of biases in the data sources.
- Difficulties in obtaining data: obtaining individual participant data (IPD) from NRSs or other RWD sources might be difficult. Reported aggregated estimates on relative effects may be biased if non-optimal analysis methods have been used. Use of IPD from NRSs was examined in the GetReal case study – Incorporating non-randomised studies in NMA of RCTs – schizophrenia.
- Reliability of results: the inclusion of NRSs or other RWD sources in the network may make the underlying assumptions of the NMA model less plausible and the NMA results less reliable.
- Increased effort to carry out: including NRSs or other RWD sources in a NMA may greatly increase the workload of the review team.
- Complex to carry out: methods for including NRSs or other RWD sources in the NMA are complex and may be difficult to implement because they require additional software expertise.
- Based on expert opinion: most approaches for including NRSs or other RWD sources require expert input, which can be time-consuming. Possible biases in the estimates from these studies may be hard to predict, either in magnitude or in direction.
What do stakeholders say?
- There may be concerns about the plausibility of the underlying assumptions of an NMA or other RWD sources when NRSs are included.
- The inclusion of NRSs or other RWD sources is seen as a threat to the validity of NMA estimates, because of the increased risk of bias in the observational estimates.
- For some stakeholders the statistical methods might be difficult to understand or carry out.
For more stakeholder feedback on specific methods examined in the GetReal case study on rheumatoid arthritis, see Using RWE to connect networks of evidence – rheumatoid arthritis.
Key contributor
Orestis Efthimiou, University of Bern and University of Ioannina School of Medicine