Effectiveness issues – Comparator

1. Trial comparators do not include current usual care or
standard of care

Current usual care (or standard of care) for the healthcare system of interest is not included as a comparator in the clinical trial. This may be because there is wide variation in usual care across healthcare systems, so that not all options can be included in a single study. A ’usual care’ medicine in the country of interest may not be licensed or reimbursed, or it may not be recommended for use (for example, in clinical guidelines) in some study countries, preventing its inclusion in the trial. For some medicines, a placebo-controlled trial may have been required to support regulatory approval, for example to resolve safety concerns, which might preclude the inclusion of usual care as a comparator in the trial. It is possible that more than one ‘usual care’ comparator is relevant for different segments of the target population, for example if a new diagnostic paradigm is involved.

In these situations, any comparison with usual care based on clinical trial data will therefore need to rely on an indirect comparison, for example a network meta-analysis based on an evidence network of results from all trials in populations with the disease of interest. Such analyses depend on statistical modelling assumptions (mostly concerning heterogeneity of the results across the source trials) as well as similarity in the design of the source trials (for example, study durations, study populations and definitions of health outcomes). Results of such meta-analyses may be associated with high levels of uncertainty. They are viewed with caution by some decision-makers because they are quite new (not yet fully ‘tried and tested’). They are also quite complex (loss of transparency) and not yet widely understood.

2. A trial-based indirect comparison is not possible

In a situation where there is no direct (head-to-head) trial comparison with usual care (or standard of care) for the healthcare system of interest, it may not be possible to construct a connected evidence network based on available trials to support any required indirect comparison with usual care. For example, there may be no trials comparing the new medicine and usual care with the same alternative therapy option (for example, placebo). It is also possible that patient groups in studies directly comparing alternative therapies are sufficiently heterogeneous to preclude the studies being combined as part of a network meta-analysis.

3. Trial participants withdraw from therapy or cross over
between treatment groups

Differential withdrawal rates between study arms in a trial may complicate interpretation of the findings, especially if there is divergence between intention-to-treat and on-treatment results. If study participants cross over between treatment groups after reaching a study endpoint (for example, disease progression in cancer) the ability of the trial to report unbiased comparisons for longer-term ‘effectiveness’ outcomes (for example, overall survival) is compromised.

4. Administration of trial comparator differs from usual practice

Although the usual care (or standard of care) medicine for the healthcare system is included as a comparator in the trial, its administration in the study (for example, dose, dose titration/escalation, frequency, route of administration, monitoring) may differ from usual practice in the healthcare system of interest. This may raise concerns about the transferability of study results to (local) usual practice. For some medicines the clinical background and skill level of the administering clinicians may be important.

5. Evidence is from single arm trials only

In some circumstances, in particular for rare diseases, accelerated approval may be sought for medicines in the absence of data from comparative randomised trials. In this situation the effectiveness of the new medicine needs to be estimated from single arm trials of the new medicine and trials or observational studies of comparator interventions.