In a situation where there is no direct (head-to-head) trial comparison with usual care (or standard of care) for the healthcare system of interest, it may not be possible to construct a connected evidence network based on available trials to support any required indirect comparison with usual care. For example, there may be no trials comparing the new medicine and usual care with the same alternative therapy option (such as placebo). It is also possible that patient groups in a study directly comparing alternative therapies are sufficiently different to preclude the study being pooled with other studies as part of a network meta-analysis.
Effectiveness challenges – Intervention / Comparator
Adherence in study differs from usual practice
Outcomes reported in trials are usually for study participants at a high level of adherence to the study medicine and comparators. However, in usual practice lower levels of adherence are expected, potentially with different levels of adherence for different therapy options resulting from differences in side effects or methods of administration. In the absence of real-world evidence, an understanding of the relationship between effectiveness and adherence is required to project estimates of effectiveness in usual practice (with sub-optimal adherence) from efficacy reported in trials. However, the relationship (often non-linear) may be difficult to predict. Having real-world data on adherence alone is insufficient to estimate effectiveness.
Adherence in study differs from usual practice
Outcomes reported in trials are usually for study participants at a high level of adherence to the study medicine and comparators. However, in usual practice lower levels of adherence are expected, potentially with different levels of adherence for different therapy options resulting from differences in side effects or methods of administration. In the absence of real-world evidence, an understanding of the relationship between effectiveness and adherence is required to project estimates of effectiveness in usual practice (with sub-optimal adherence) from efficacy reported in trials. However, the relationship (often non-linear) may be difficult to predict. Having real-world data on adherence alone is insufficient to estimate effectiveness.
Administration of therapy is inconsistent with usual practice
The administration of the study medicine in trials (for example, dose, dose titration/escalation, frequency, route of administration, monitoring) may differ from the schedule that is likely to be used in clinical practice. This is perhaps more likely if the new therapy is added to current usual care, or if the treatment itself may be intentionally misused by patients if its administration is not under strict control (for example, opioids in pain relief). In some cases the clinical background and skill level of the administering clinicians may be important.
Administration of therapy is inconsistent with usual practice
The administration of the study medicine in trials (for example, dose, dose titration/escalation, frequency, route of administration, monitoring) may differ from the schedule that is likely to be used in clinical practice. This is perhaps more likely if the new therapy is added to current usual care, or if the treatment itself may be intentionally misused by patients if its administration is not under strict control (for example, opioids in pain relief). In some cases the clinical background and skill level of the administering clinicians may be important.
Administration of trial comparator differs from usual practice
Although the usual care (or standard of care) medicine for the healthcare system is included as a comparator in the trial, its administration in the study (for example, dose, dose titration/escalation, frequency, route of administration, monitoring) may differ from usual practice in the country of interest. This may raise concerns about the transferability of study results to (local) usual practice. In some cases the clinical background and skill level of the administering clinicians may be important.
Administration of trial comparator differs from usual practice
Although the usual care (or standard of care) medicine for the healthcare system is included as a comparator in the trial, its administration in the study (for example, dose, dose titration/escalation, frequency, route of administration, monitoring) may differ from usual practice in the country of interest. This may raise concerns about the transferability of study results to (local) usual practice. In some cases the clinical background and skill level of the administering clinicians may be important.
Evidence available is from single arm trials only
In some circumstances, in particular for rare diseases, accelerated approval may be sought for medicines in the absence of data from comparative randomised trials. In this situation the effectiveness of the new medicine needs to be estimated from single arm trials of the new medicine and trials or observational studies of comparator interventions.
Population-adjusted indirect comparisons (a type of standardisation), have been developed to map treatment effects observed in one population into effects that would be observed in another population. Matching-Adjusted Indirect Comparison (MAIC, based on propensity score weighting) and Simulated Treatment Comparison (STC, based on outcome regression) use individual patient data (IPD) from one study to adjust for between-study differences in the distribution of variables that influence outcome. ‘Unanchored’ comparisons are required when considering single arm studies as there is no common comparator across studies. Although these methods are superior to naïve comparisons (e.g. with historical ‘controls’) they require strong assumptions about the presence of all effect modifiers and prognostic variables in the data. Their results need to be treated with some caution as an unknown amount of residual bias may remain in the statistically modelled comparisons.
These methods were not reviewed explicitly by the GetReal project. Further information can be found in NICE DSU Technical Support Document 18 (Phillippo, 2016) and this article published in Value in Health (Signorovitch, 2012).
Stopping rules for therapy are unclear
The administration of the study medicine in trials (for example, dose, dose titration/escalation, frequency, route of administration, monitoring) may differ from the schedule that is likely to be used in clinical practice. This is perhaps more likely if the new therapy is added to current usual care, or if the treatment itself may be intentionally misused by patients if its administration is not under strict control (for example, opioids in pain relief). In some cases the clinical background and skill level of the administering clinicians may be important.
Stopping rules for therapy are unclear
The administration of the study medicine in trials (for example, dose, dose titration/escalation, frequency, route of administration, monitoring) may differ from the schedule that is likely to be used in clinical practice. This is perhaps more likely if the new therapy is added to current usual care, or if the treatment itself may be intentionally misused by patients if its administration is not under strict control (for example, opioids in pain relief). In some cases the clinical background and skill level of the administering clinicians may be important.
Trial comparators do not include current usual care or standard of care
Current usual care (or standard of care) for the healthcare system of interest is not included as a comparator in the clinical trial. This may be because there is wide variation in usual care across healthcare systems, so that not all options can be included in a single study. A ’usual care’ medicine in the country of interest may not be licensed or reimbursed, or it may not be recommended for use (for example, in clinical guidelines) in some study countries, preventing its inclusion in the trial. In some cases a placebo-controlled trial may have been required to support regulatory approval, for example to resolve safety concerns, which might preclude the inclusion of usual care as a comparator in the trial. It is possible that more than one usual care comparator is relevant for different segments of the target population, for example if a new diagnostic paradigm is involved.
Any comparison with usual care based on clinical trial data will therefore need to rely on an indirect comparison, for example a network meta-analysis based on an evidence network of results from all trials in populations with the disease of interest. Such analyses depend on statistical modelling assumptions (mostly concerning heterogeneity of the results across the source trials) as well as similarity in the design of the source trials (for example, study durations, study populations and definitions of health outcomes). Results of such meta-analyses may be associated with high levels of uncertainty. They are viewed with caution by some decision-makers because they are quite new (not yet fully ‘tried and tested’), are quire complex (loss of transparency) and are not yet widely understood.
Trial comparators do not include current usual care or standard of care
Current usual care (or standard of care) for the healthcare system of interest is not included as a comparator in the clinical trial. This may be because there is wide variation in usual care across healthcare systems, so that not all options can be included in a single study. A ’usual care’ medicine in the country of interest may not be licensed or reimbursed, or it may not be recommended for use (for example, in clinical guidelines) in some study countries, preventing its inclusion in the trial. In some cases a placebo-controlled trial may have been required to support regulatory approval, for example to resolve safety concerns, which might preclude the inclusion of usual care as a comparator in the trial. It is possible that more than one usual care comparator is relevant for different segments of the target population, for example if a new diagnostic paradigm is involved.
Any comparison with usual care based on clinical trial data will therefore need to rely on an indirect comparison, for example a network meta-analysis based on an evidence network of results from all trials in populations with the disease of interest. Such analyses depend on statistical modelling assumptions (mostly concerning heterogeneity of the results across the source trials) as well as similarity in the design of the source trials (for example, study durations, study populations and definitions of health outcomes). Results of such meta-analyses may be associated with high levels of uncertainty. They are viewed with caution by some decision-makers because they are quite new (not yet fully ‘tried and tested’), are quire complex (loss of transparency) and are not yet widely understood.
Trial comparators do not include current usual care or standard of care
Current usual care (or standard of care) for the healthcare system of interest is not included as a comparator in the clinical trial. This may be because there is wide variation in usual care across healthcare systems, so that not all options can be included in a single study. A ’usual care’ medicine in the country of interest may not be licensed or reimbursed, or it may not be recommended for use (for example, in clinical guidelines) in some study countries, preventing its inclusion in the trial. In some cases a placebo-controlled trial may have been required to support regulatory approval, for example to resolve safety concerns, which might preclude the inclusion of usual care as a comparator in the trial. It is possible that more than one usual care comparator is relevant for different segments of the target population, for example if a new diagnostic paradigm is involved.
Any comparison with usual care based on clinical trial data will therefore need to rely on an indirect comparison, for example a network meta-analysis based on an evidence network of results from all trials in populations with the disease of interest. Such analyses depend on statistical modelling assumptions (mostly concerning heterogeneity of the results across the source trials) as well as similarity in the design of the source trials (for example, study durations, study populations and definitions of health outcomes). Results of such meta-analyses may be associated with high levels of uncertainty. They are viewed with caution by some decision-makers because they are quite new (not yet fully ‘tried and tested’), are quire complex (loss of transparency) and are not yet widely understood.
Trial participants withdraw from therapy or cross over between treatment groups
Differential withdrawal rates between study arms in a trial may complicate interpretation of the findings, especially if there is divergence between intention-to-treat and on-treatment results. If study participants cross over between treatment groups after reaching a study endpoint (for example, disease progression in cancer) the ability of the trial to report unbiased comparisons for longer-term ‘effectiveness’ outcomes (for example, overall survival) is compromised.