Overview of methods for predicting outcomes, including approaches to bridge the efficacy-effectiveness gap

What is it?

The generalisability of results observed in a randomised controlled trial (RCT) to real-life settings is a fundamental issue for pharmaceutical research and development (R&D), regulatory, health technology assessment (HTA) and reimbursement agencies. The potential difference between RCT outcomes and effects in real-world settings has been called the efficacy–effectiveness gap (for a definition, see here). A GetReal literature review revealed a number of methods that have been developed to predict the effectiveness of treatments in the real-world, based on available RCT data, and which can bridge the efficacy-effectiveness gap (the articles identified in this review can be found here). Such methods include:

  • multi-state models
  • discrete event simulation models
  • physiology-based models
  • survival and generalized linear models.

Outcomes are predicted over time for new patient populations or for drug doses not tested in clinical trials. Software used to perform predictions include Excel, SPSS, C++ and ARENA (Panayidou et al 2016).

Figure. Use of modelling to extend RCT data.AM-l-t.0_

Several further methods have been explored by GetReal, which are illustrated by the case studies described in the table below.

Table. Use of modelling to extend RCT data: description of method and GetReal experience.

Method GetReal case study
Prediction for new population
Prior-to-launch prediction of treatment effectiveness based on RCT efficacy data, learning treatment decision from observational data (see description here) Rheumatoid arthritis
Adjusting real-world evidence for bias to produce relative effectiveness estimates (using regression-based adjustment: parametric regression based on matched sample, instrumental variables, multivariate adjustment, doubly robust methods) (see description here) Rheumatoid arthritis
Reweighting clinical trial data to mimic (extrapolate to) real-world population: using propensity score reweighting or inverse probability weighting (see description of propensity score reweighting here and inverse probability weighting here) Non-small cell lung cancer
Prediction over time
Predicting overall or progression-free survival for a longer time horizon, using RCT and observational data (see description here for the method used in non-small cell lung cancer and here for the method used in metastatic melanoma) Non-small cell lung cancer and
Metastatic melanoma

A questionnaire to assess the relevance and credibility of modelling studies has been developed through a collaboration between the International Society of Pharmacoeconomics and Outcome Research (ISPOR), the Academy of Managed Care (AMCP) and the National Pharmaceutical Council (NPS). The National Institute for Health and Care Excellence (NICE) Decision Support Unit provides general guidance on individual patient simulations.

Why is it useful?

It is essential for stakeholders, such as pharmaceutical R&D, regulatory, HTA and reimbursement agencies, to know the effectiveness of a treatment at launch. As evidence on the effect of a treatment in the real-world is lacking when the treatment enters the market, methods to predict effectiveness can offer insights beyond the available RCT evidence to support decision-making at this stage, for example:

  • Pharmaceutical R&D: simulate future clinical trials.
  • Regulatory agencies: decide on licencing of treatment for broader disease population and treatment period.
  • HTA and reimbursement agencies: predict cost-effectiveness of the new treatment.

When is it useful?

  • Lack of generalisability: when treatment results from RCT conditions (efficacy) are not generalisable to a real-world setting (i.e. there is an efficacy-effectiveness gap). This might be the case if:
    • the RCTs were conducted in very specific populations that exclude a proportion of the usual disease population, for example older patients or patients with severe disease or comorbidities
    • the RCTs were conducted over short time periods only.
  • Pre-launch: predicting effectiveness of treatments before a medicine is launched is valuable for stakeholders.

What are the limitations?

  • Validation of the prediction is difficult to perform.
  • Availability of individual participant data (IPD): for some prediction models, good-quality IPD from RCTs and observational studies need to be available.

What do stakeholders say?

Stakeholder views about the use of modelling to predict relative effectiveness from real-world data (RWD) were sought through stakeholder interviews. The following concerns were raised:

  • transparency and reproducibility
  • model assumptions that have large impact to output
  • predicting relative effectiveness based on RWD is still an unknown area
  • concept of using RWD to model effectiveness is unconvincing.

Key contributors

Noemmi Hummel, University of Bern
Eva-Maria Didden, University of Bern