Methods for predicting outcomes

What is it?

The generalisability of results observed in a randomised controlled trial (RCT) to real-life settings is a fundamental issue for pharmaceutical research and development (R&D), regulatory, health technology assessment (HTA) and reimbursement agencies. The potential difference between RCT outcomes and effects in real-world settings has been called the efficacy–effectiveness gap (for a definition, see Clarify the Issues).

A GetReal literature review (Panayidou et al, 2016) revealed a number of methods that have been developed to predict the effectiveness of treatments in the real-world, based on available RCT data, which can bridge the efficacy-effectiveness gap (the articles identified in this review can be found at this GetReal Zotero page). Such methods include:

  • multi-state models
  • discrete event simulation models
  • physiology-based models
  • survival and generalized linear models.

Outcomes are predicted over time for new patient populations or for drug doses not tested in clinical trials. Software used to perform predictions include Excel, SPSS, C++ and ARENA (Panayidou et al, 2016).

Several further methods have been explored by GetReal, which are illustrated by the case studies described in the table below.

Table. Use of modelling to extend RCT data: description of method and GetReal experience.

Method GetReal case study
Prediction for new population
Prior-to-launch prediction of treatment effectiveness based on RCT efficacy data, learning treatment decision from observational data

For a description, see Predicting Effectiveness based on RCT Efficacy Data and RWE before Launch

 See Predicting Effectiveness based on RCT Efficacy Data and RWE – Rheumatoid Arthritis
Adjusting real-world evidence for bias to produce relative effectiveness estimates using regression-based adjustment: parametric regression based on matched sample, instrumental variables, multivariate adjustment, doubly robust methods

For a description, see Adjusting for Bias

 See Using RWE to Connect Networks of Evidence- Rheumatoid Arthritis
Reweighting clinical trial data to mimic (extrapolate to) real-world population using propensity score reweighting or inverse probability weighting

For a description of propensity score reweighting, see Propensity Weighting to Generate Estimates of Relative Effectiveness from Trials and for inverse probability weighting, see Adjusting for Bias

 See Propensity Weighting and Extrapolation- Non-Small-Cell Lung Cancer
Prediction over time
Predicting overall or progression-free survival for a longer time horizon, using RCT and observational data

See Model-Averaging Approach to Predicting Long-Term Effectiveness for the method used in the NSCLC case study

See Using RWE to Inform Relative Effectiveness Estimates and Trial Design – Metastatic Melanoma for the method used in metastatic melanoma

 See Propensity Weighting and Extrapolation – Non-Small-Cell Lung Cancer

See Using RWE to Inform Relative Effectiveness Estimates and Trial Design – Metastatic Melanoma

A questionnaire to assess the relevance and credibility of modelling studies was developed through a collaboration between the International Society of Pharmacoeconomics and Outcome Research (ISPOR), the Academy of Managed Care (AMCP) and the National Pharmaceutical Council (NPS) (Caro et al, 2014). The National Institute for Health and Care Excellence (NICE) Decision Support Unit provides general guidance on individual patient simulations (Technical Support Document 15, NICE DSU).

Why is it useful?

It is essential for stakeholders, such as pharmaceutical R&D, regulatory, HTA and reimbursement agencies, to know the effectiveness of a treatment at launch. As evidence on the effect of a treatment in the real-world is lacking when the treatment enters the market, methods to predict effectiveness can offer insights beyond the available RCT evidence to support decision-making at this stage, for example:

  • Pharmaceutical R&D: simulate future clinical trials.
  • Regulatory agencies: decide on licencing of treatment for broader disease population and treatment period.
  • HTA and reimbursement agencies: predict clinical effectiveness and cost-effectiveness of the new treatment.

When is it useful?

  • Lack of generalisability: when treatment results from RCT conditions (efficacy) are not generalisable to a real-world setting (i.e. there is an efficacy-effectiveness gap). This might be the case if:
    • the RCTs were conducted in very specific populations that exclude a proportion of the usual disease population, for example older patients or patients with severe disease or comorbidities
    • the RCTs were conducted over short time periods only.
  • Pre-launch: predicting effectiveness of treatments before a medicine is launched is valuable for stakeholders.

What are the limitations?

  • Validation of the prediction is difficult to perform.
  • Availability of individual participant data (IPD): for some prediction models, good-quality IPD from RCTs and observational studies need to be available.

What do stakeholders say?

Stakeholder views about the use of modelling to predict relative effectiveness from real-world data (RWD) were sought through stakeholder interviews. The following concerns were raised:

  • transparency and reproducibility
  • model assumptions that have large impact to output
  • predicting relative effectiveness based on RWD is still an unknown area
  • concept of using RWD to model effectiveness is unconvincing.

Key contributors

Noemmi Hummel, University of Bern
Eva-Maria Didden, University of Bern