A trial-based indirect comparison is not possible

In a situation where there is no direct (head-to-head) trial comparison with usual care (or standard of care) for the healthcare system of interest, it may not be possible to construct a connected evidence network based on available trials to support any required indirect comparison with usual care. For example, there may be no trials comparing the new medicine and usual care with the same alternative therapy option (such as placebo). It is also possible that patient groups in a study directly comparing alternative therapies are sufficiently different to preclude the study being pooled with other studies as part of a network meta-analysis.

 

Adherence in study differs from usual practice

Outcomes reported in trials are usually for study participants at a high level of adherence to the study medicine and comparators. However, in usual practice lower levels of adherence are expected, potentially with different levels of adherence for different therapy options resulting from differences in side effects or methods of administration. In the absence of real-world evidence, an understanding of the relationship between effectiveness and adherence is required to project estimates of effectiveness in usual practice (with sub-optimal adherence) from efficacy reported in trials. However, the relationship (often non-linear) may be difficult to predict. Having real-world data on adherence alone is insufficient to estimate effectiveness.

 

Adherence in study differs from usual practice

Outcomes reported in trials are usually for study participants at a high level of adherence to the study medicine and comparators. However, in usual practice lower levels of adherence are expected, potentially with different levels of adherence for different therapy options resulting from differences in side effects or methods of administration. In the absence of real-world evidence, an understanding of the relationship between effectiveness and adherence is required to project estimates of effectiveness in usual practice (with sub-optimal adherence) from efficacy reported in trials. However, the relationship (often non-linear) may be difficult to predict. Having real-world data on adherence alone is insufficient to estimate effectiveness.

 

Administration of therapy is inconsistent with usual practice

The administration of the study medicine in trials (for example, dose, dose titration/escalation, frequency, route of administration, monitoring) may differ from the schedule that is likely to be used in clinical practice. This is perhaps more likely if the new therapy is added to current usual care, or if the treatment itself may be intentionally misused by patients if its administration is not under strict control (for example, opioids in pain relief). In some cases the clinical background and skill level of the administering clinicians may be important.

 

Administration of therapy is inconsistent with usual practice

The administration of the study medicine in trials (for example, dose, dose titration/escalation, frequency, route of administration, monitoring) may differ from the schedule that is likely to be used in clinical practice. This is perhaps more likely if the new therapy is added to current usual care, or if the treatment itself may be intentionally misused by patients if its administration is not under strict control (for example, opioids in pain relief). In some cases the clinical background and skill level of the administering clinicians may be important.

 

Administration of trial comparator differs from usual practice

Although the usual care (or standard of care) medicine for the healthcare system is included as a comparator in the trial, its administration in the study (for example, dose, dose titration/escalation, frequency, route of administration, monitoring) may differ from usual practice in the country of interest. This may raise concerns about the transferability of study results to (local) usual practice. In some cases the clinical background and skill level of the administering clinicians may be important.

 

Administration of trial comparator differs from usual practice

Although the usual care (or standard of care) medicine for the healthcare system is included as a comparator in the trial, its administration in the study (for example, dose, dose titration/escalation, frequency, route of administration, monitoring) may differ from usual practice in the country of interest. This may raise concerns about the transferability of study results to (local) usual practice. In some cases the clinical background and skill level of the administering clinicians may be important.

 

Evidence available is from single arm trials only

 

Stopping rules for therapy are unclear

The administration of the study medicine in trials (for example, dose, dose titration/escalation, frequency, route of administration, monitoring) may differ from the schedule that is likely to be used in clinical practice. This is perhaps more likely if the new therapy is added to current usual care, or if the treatment itself may be intentionally misused by patients if its administration is not under strict control (for example, opioids in pain relief). In some cases the clinical background and skill level of the administering clinicians may be important.

 

Stopping rules for therapy are unclear

The administration of the study medicine in trials (for example, dose, dose titration/escalation, frequency, route of administration, monitoring) may differ from the schedule that is likely to be used in clinical practice. This is perhaps more likely if the new therapy is added to current usual care, or if the treatment itself may be intentionally misused by patients if its administration is not under strict control (for example, opioids in pain relief). In some cases the clinical background and skill level of the administering clinicians may be important.

 

Trial comparators do not include current usual care or standard of care

Current usual care (or standard of care) for the healthcare system of interest is not included as a comparator in the clinical trial. This may be because there is wide variation in usual care across healthcare systems, so that not all options can be included in a single study. A ’usual care’ medicine in the country of interest may not be licensed or reimbursed, or it may not be recommended for use (for example, in clinical guidelines) in some study countries, preventing its inclusion in the trial. In some cases a placebo-controlled trial may have been required to support regulatory approval, for example to resolve safety concerns, which might preclude the inclusion of usual care as a comparator in the trial. It is possible that more than one usual care comparator is relevant for different segments of the target population, for example if a new diagnostic paradigm is involved.

Any comparison with usual care based on clinical trial data will therefore need to rely on an indirect comparison, for example a network meta-analysis based on an evidence network of results from all trials in populations with the disease of interest. Such analyses depend on statistical modelling assumptions (mostly concerning heterogeneity of the results across the source trials) as well as similarity in the design of the source trials (for example, study durations, study populations and definitions of health outcomes). Results of such meta-analyses may be associated with high levels of uncertainty. They are viewed with caution by some decision-makers because they are quite new (not yet fully ‘tried and tested’), are quire complex (loss of transparency) and are not yet widely understood.

 

Trial comparators do not include current usual care or standard of care

Current usual care (or standard of care) for the healthcare system of interest is not included as a comparator in the clinical trial. This may be because there is wide variation in usual care across healthcare systems, so that not all options can be included in a single study. A ’usual care’ medicine in the country of interest may not be licensed or reimbursed, or it may not be recommended for use (for example, in clinical guidelines) in some study countries, preventing its inclusion in the trial. In some cases a placebo-controlled trial may have been required to support regulatory approval, for example to resolve safety concerns, which might preclude the inclusion of usual care as a comparator in the trial. It is possible that more than one usual care comparator is relevant for different segments of the target population, for example if a new diagnostic paradigm is involved.

Any comparison with usual care based on clinical trial data will therefore need to rely on an indirect comparison, for example a network meta-analysis based on an evidence network of results from all trials in populations with the disease of interest. Such analyses depend on statistical modelling assumptions (mostly concerning heterogeneity of the results across the source trials) as well as similarity in the design of the source trials (for example, study durations, study populations and definitions of health outcomes). Results of such meta-analyses may be associated with high levels of uncertainty. They are viewed with caution by some decision-makers because they are quite new (not yet fully ‘tried and tested’), are quire complex (loss of transparency) and are not yet widely understood.

 

Trial participants withdraw from therapy or cross over between treatment groups

Differential withdrawal rates between study arms in a trial may complicate interpretation of the findings, especially if there is divergence between intention-to-treat and on-treatment results. If study participants cross over between treatment groups after reaching a study endpoint (for example, disease progression in cancer) the ability of the trial to report unbiased comparisons for longer-term ‘effectiveness’ outcomes (for example, overall survival) is compromised.