Effectiveness issues – Comparator

1. Trial comparators do not include current usual care or standard of care

Current usual care (or standard of care) for the healthcare system of interest is not included as a comparator in the clinical trial. This may be because there is wide variation in usual care across healthcare systems, so that not all options can be included in a single study. A ’usual care’ medicine in the country of interest may not be licensed or reimbursed, or it may not be recommended for use (for example, in clinical guidelines) in some study countries, preventing its inclusion in the trial. In some cases a placebo-controlled trial may have been required to support regulatory approval, for example to resolve safety concerns, which might preclude the inclusion of usual care as a comparator in the trial. It is possible that more than one usual care comparator is relevant for different segments of the target population, for example if a new diagnostic paradigm is involved.

Any comparison with usual care based on clinical trial data will therefore need to rely on an indirect comparison, for example a network meta-analysis based on an evidence network of results from all trials in populations with the disease of interest. Such analyses depend on statistical modelling assumptions (mostly concerning heterogeneity of the results across the source trials) as well as similarity in the design of the source trials (for example, study durations, study populations and definitions of health outcomes). Results of such meta-analyses may be associated with high levels of uncertainty. They are viewed with caution by some decision-makers because they are quite new (not yet fully ‘tried and tested’), are quire complex (loss of transparency) and are not yet widely understood.

2. A trial-based indirect comparison is not possible

In a situation where there is no direct (head-to-head) trial comparison with usual care (or standard of care) for the healthcare system of interest, it may not be possible to construct a connected evidence network based on available trials to support any required indirect comparison with usual care. For example, there may be no trials comparing the new medicine and usual care with the same alternative therapy option (such as placebo). It is also possible that patient groups in a study directly comparing alternative therapies are sufficiently different to preclude the study being pooled with other studies as part of a network meta-analysis.

3. Trial participants withdraw from therapy or cross over between treatment groups

Differential withdrawal rates between study arms in a trial may complicate interpretation of the findings, especially if there is divergence between intention-to-treat and on-treatment results. If study participants cross over between treatment groups after reaching a study endpoint (for example, disease progression in cancer) the ability of the trial to report unbiased comparisons for longer-term ‘effectiveness’ outcomes (for example, overall survival) is compromised.

4. Administration of trial comparator differs from usual practice

Although the usual care (or standard of care) medicine for the healthcare system is included as a comparator in the trial, its administration in the study (for example, dose, dose titration/escalation, frequency, route of administration, monitoring) may differ from usual practice in the country of interest. This may raise concerns about the transferability of study results to (local) usual practice. In some cases the clinical background and skill level of the administering clinicians may be important.