Predicting effectiveness based on RCT efficacy data and RWE before launch

What is it?

Predicting the real-world effect of a new treatment before launch, i.e. early in the pharmaceutical research and development (R&D) process, can be done by combining trial results on the efficacy of the new treatment with observational evidence on treatment choice and clinical expert advice. The method described here comprises two modelling stages; the first identifies patients who are likely to receive the new treatment in usual clinical practice, and the second predicts treatment effect in these patients:

  • First modelling stage – predicting treatment decision: advice from clinical experts is important to decide whether a specific patient is likely to receive the new treatment. This should be discussed with clinicians when using the suggested predictive modelling strategy (see Box 1 for examples of questions to ask). The strategy is based on the assumption that a drug similar to the new drug is already launched and that observational data on this similar drug are available. Treatment decision rules can thus be estimated from the observational data by including appropriate treatment predictors (i.e. the covariate driving treatment choice) into a treatment decision model. Both the identification of the similar treatment and the search for important treatment predictors should be supported by expert advice (see Box 1, questions 1 and 2).
  • Second modelling stage – predicting treatment outcome and effect: a suitable treatment effect model is needed to predict the benefit from the new treatment. Involving clinicians in the choice of an appropriate measure of treatment outcome and in the selection of important prognostic factors and effect modifiers (see Box 1, questions 3 and 4), prevents a purely statistical view on the research problem and the risk of modelling bias decreases. Using this approach, the prediction of treatment effect and outcome is informed by randomised controlled trial (RCT) evidence on the relative effect of the new drug and all important effect-modifying impacts, and by RCT and real-world evidence (RWE) on the effects of the prognostic factors.

Variable selection:
Deciding which covariates to include in the prediction model is not a trivial task. The use of expert advice should be combined with the appropriate use of the least absolute shrinkage and selection operator (LASSO). This method forces some of the unknown covariate effects to be set to zero, in other words, some of the potential treatment predictors, prognostic factors and effect modifiers are automatically excluded from the prediction model, unless they have been selected based on expert opinion. The amount of shrinkage is optimised based on leave-one-out cross-validation, also known as the Jackknife method.

Box 1: Questions for clinicians when using a prediction model

  1. Is there any market-approved treatment that is comparable to the new therapy?
  2. Which patient characteristics and disease severity measures make clinicians decide for or against the existing similar treatment?
  3. What is a reliable and frequently reported measure of treatment outcome?
  4. Which patient and disease characteristics should necessarily be considered as predictors of treatment effect?
    – Which of these covariates are assumed to act as prognostic factors?
    – Which of these covariates are potential effect modifiers?

‘Effectiveness challenge(s)’ addressed by the method

This mathematical modelling approach allows the real-world treatment effect to be predicted from the trial efficacy data, while accounting for a possible efficacy-effectiveness gap (for a definition, see here). Typically, such a gap might emerge due to a number of factors, such as differences between RCT and real-world patient populations, randomisation vs. drug administration guidelines, and/or adherence issues. To address these potential inconsistencies, this prediction model is informed by RWE on the effectiveness of a drug similar to the drug under investigation, and by the individual and disease characteristics of those patients receiving the similar treatment.

When is it useful?

This method should be considered when an efficacy-effectiveness gap is expected between RCT results and the effects of the medicine in the real-world. It has been developed particularly for use early in the pharmaceutical R&D process, and is most suitable when results of phase 2 or phase 3 trials are available. However, in theory, predictions can be made for any population of interest.

What are the limitations?

  • Observational studies on a similar treatment are needed: observational studies on the effectiveness of a treatment similar to the new treatment must be available, and patients receiving this similar treatment should also be likely to receive the new medicine.
  • Individual participant data (IPD) is needed: IPD from RCTs and particularly from observational studies is needed. Missing data and inconsistencies in reporting are common problems.
  • Effect of different variable selection techniques: the use of different methods for variable selection will usually yield different results. There is no unique solution. However, automatic selection procedures, such as the LASSO, are preferred over step-wise selection and elimination. Sensitivity analyses are recommended.
  • External/predictive validation: this is only possible in retrospective analyses.
  • Different trial and real-world patient populations: this method may not be appropriate if the trial population is too different from a real-world patient population.

What do stakeholders say?

  • Stakeholders would be interested to see how sensitive the model is to differences between trial and real-world populations. It was felt that sub-group analyses may result in instructive conclusions.
  • This modelling approach may provide information on how RCT inclusion criteria could be specified in a more real-world oriented manner and on how clinical trials could be designed more pragmatically.

Key contributor

Eva-Maria Didden, University of Bern