1. Trial outcomes are not considered to be measures of effectiveness
Outcomes reported in pivotal trials may not be considered to be measures of relative effectiveness from a health technology assessment (HTA) or reimbursement perspective. These outcomes (efficacy or safety) are selected to meet the needs of regulatory approval, but may not be optimal for some HTA agencies. A number of factors may be relevant. Trial outcomes may represent physiological parameters, such as tumour response, blood haemoglobin level or lung function, which are not considered to be patient-relevant. However, these may serve as surrogate endpoints (proxies) for effectiveness outcomes of relevance to HTA, but the relationship between the surrogate and ‘final’ endpoint needs to be demonstrated quantitatively. Outcomes that are clinically assessed disease activity indices may be considered measures of effectiveness if they are validated and are widely used. Outcomes that are composite endpoints (for example, MACE: major adverse cardiac events) may need to be disaggregated into their components for consideration in HTA.
2. There is uncertainty in reported trial outcomes
Trial results may be difficult to interpret because of large uncertainty (wide confidence intervals) in the outcome measures. Trials may not have been powered specifically to detect differences in patient-relevant endpoints such as health-related quality of life. In some healthcare systems, secondary and tertiary outcomes may be considered to be lower quality evidence. High levels of uncertainty may be reported if there are lower rates of the outcome event than expected. Results for subgroups of importance to the health system of interest will have greater uncertainty, and in some cases may not have been reported.
3. There are inconsistent results across or within trials
Trial results may be difficult to interpret because of different (possibly inconsistent) efficacy results across the pivotal studies. This may be due to differences in the trial populations associated with differences in efficacy across important sub-populations.
Results for an individual trial may vary in magnitude and/or direction for different outcomes, making interpretation less straightforward, for example when reviewing results for individual components of composite endpoints.
4. Definition of trial outcome is inconsistent across studies or with usual practice
The definition of an outcome used in the trial may be inconsistent with that used in other studies of usual care (or standard of care) or other comparator therapies. This may be for a number of reasons: lack of standard outcomes in a disease area, regional variations in what is considered important for good health, variations in the availability or use of study instruments (for example, patient-reported outcomes may not have been validated and versions may not be available in all languages), or definitions of outcomes changing over time. This may limit the ability to combine the results of studies in systematic reviews and meta-analyses.
In addition, the definition of outcomes used in the trial may differ from the definitions used in usual practice. The context of a clinical study (where patients are assessed more intensively and frequently, and there is access to a wider range of diagnostic services) may enable outcomes to be defined and measured in a way that would not be feasible in usual practice.
5. There is a high risk of biased comparisons from observational (non-randomised) data
During medicine development, observational data are generally not available on effectiveness of the new therapy. They are however used for a number of purposes: to describe the natural history of disease, disease burden and treatment patterns, or the relationship between surrogate and final endpoints, or to validate new study endpoints or provide information on the effectiveness of comparator therapies. Some of this information may be used indirectly in estimating the effectiveness of the new therapy, for example through predictive modelling of long-term outcomes. Assessors will be particularly interested in the generalisability of the study population and the statistical methods used to control for bias.
In situations of conditional reimbursement, potentially within an adaptive pathway for a new medicine, observational data (for example, from registries) on the effectiveness of new medicine may be presented and assessed at HTA reviews. Comparisons between therapy alternatives of health outcomes based on non-randomised studies are particularly subject to bias: careful study design is required to minimise the bias. A variety of analytical techniques are available to adjust for imbalances observed between study groups that may affect the comparison, although this is unlikely to fully eliminate bias.
6. Modelling of final outcomes from trial efficacy is not robust
Evidence of relative effectiveness derived from modelling final outcomes (effectiveness) from trial outcomes (efficacy) may be considered weak or unacceptable. This may be because the association between final (model) and surrogate (trial) and final outcomes is weak, and therefore estimated with low confidence (wide uncertainty). More technically, the relationship may be poorly specified statistically, or not well validated. The modelling approach itself may be inadequately described or justified. The data sources (especially sources other than trials) may not be considered relevant to the population under consideration, or are of poor quality (missing data, potential biases in their analysis). In some healthcare systems the use of modelling per se may be inadmissible or considered to be (only) supportive evidence.