Trial treatment pathway is not generalisable to usual practice

‘Treatment pathway’ refers to the sequence of previous treatments received, based on patient selection criteria and response to previous therapies, often reflected in clinical guidelines. Assessment of the position in the treatment pathway drives the definition of the treatment population and choice of appropriate comparator. This pathway may have changed during the course of the trial if a new medicine has been approved or new clinical guidelines have been implemented locally. In some cases, the introduction of the medicine of interest will itself alter the treatment pathway. Alternatively there may not yet be an established treatment pathway or sequencing, and as a result there may be variations across countries or health centres within countries.

Trial comparators do not include current usual care or standard of care

Current usual care (or standard of care) for the healthcare system of interest is not included as a comparator in the clinical trial. This may be because there is wide variation in usual care across healthcare systems, so that not all options can be included in a single study. A ’usual care’ medicine in the country of interest may not be licensed or reimbursed, or it may not be recommended for use (for example, in clinical guidelines) in some study countries, preventing its inclusion in the trial. In some cases a placebo-controlled trial may have been required to support regulatory approval, for example to resolve safety concerns, which might preclude the inclusion of usual care as a comparator in the trial. It is possible that more than one usual care comparator is relevant for different segments of the target population, for example if a new diagnostic paradigm is involved.

Any comparison with usual care based on clinical trial data will therefore need to rely on an indirect comparison, for example a network meta-analysis based on an evidence network of results from all trials in populations with the disease of interest. Such analyses depend on statistical modelling assumptions (mostly concerning heterogeneity of the results across the source trials) as well as similarity in the design of the source trials (for example, study durations, study populations and definitions of health outcomes). Results of such meta-analyses may be associated with high levels of uncertainty. They are viewed with caution by some decision-makers because they are quite new (not yet fully ‘tried and tested’), are quire complex (loss of transparency) and are not yet widely understood.

Trial participants are at a different position in treatment pathway

‘Treatment pathway’ refers to the sequence of previous treatments received, based on patient selection criteria and response to previous therapies, often reflected in clinical guidelines. The position of patients in the treatment pathway drives the choice of appropriate comparator.

The position in the treatment pathway occupied by trial participants may differ from that which they would occupy in usual practice in the healthcare system of interest. For example, in usual practice the new medicine may be considered for use (only) after the disease progressed on two types of therapy, whereas trial participants may have experienced disease progression on just one. This assumes that the pathway has not changed during the course of the trial.

There is a high risk of biased comparisons from observational (non-randomised) data

During medicine development, observational data are generally not available on effectiveness of the new therapy. They are however used for a number of purposes: to describe the natural history of disease, disease burden and treatment patterns, or the relationship between surrogate and final endpoints, or to validate new study endpoints or provide information on the effectiveness of comparator therapies. Some of this information may be used indirectly in estimating the effectiveness of the new therapy, for example through predictive modelling of long-term outcomes. Assessors will be particularly interested in the generalisability of the study population and the statistical methods used to control for bias.

In situations of conditional reimbursement, potentially within an adaptive pathway for a new medicine, observational data (for example, from registries) on the effectiveness of new medicine may be presented and assessed at HTA reviews. Comparisons between therapy alternatives of health outcomes based on non-randomised studies are particularly subject to bias: careful study design is required to minimise the bias. A variety of analytical techniques are available to adjust for imbalances observed between study groups that may affect the comparison, although this is unlikely to fully eliminate bias.

Trial outcomes not considered to be measures of effectiveness

Outcomes reported in pivotal trials may not be considered to be measures of relative effectiveness from a health technology assessment (HTA) or reimbursement perspective. These outcomes (efficacy or safety) are selected to meet the needs of regulatory approval, but may not be optimal for some HTA agencies. A number of factors may be relevant. Trial outcomes may represent physiological parameters, such as tumour response, blood haemoglobin level or lung function, which are not considered to be patient-relevant. However, these may serve as surrogate endpoints (proxies) for effectiveness outcomes of relevance to HTA, but the relationship between the surrogate and ‘final’ endpoint needs to be demonstrated quantitatively. Outcomes that are clinically assessed disease activity indices may be considered measures of effectiveness if they are validated and are widely used. Outcomes that are composite endpoints (for example, MACE: major adverse cardiac events) may need to be disaggregated into their components for consideration in HTA.

Disease area is not well defined

The disease/indication may have an imprecise definition, and so there may be no well-established criteria for defining the population of interest (likely to receive the new intervention). A clinical code (ICD or other) may not yet have been established for the disease entity. Correspondingly there may be a lack of clinical guidelines, and usual care (or standard of care) may not be well established for the target population within an agreed treatment pathway. This may lead to uncertainty in applying trial results to the local context, as well as subsequent difficulties in setting up studies to monitor the usage and effectiveness of the medicine post-launch.