1. Trial treatment pathway is not generalisable to usual practice
‘Treatment pathway’ refers to the sequence of previous treatments received, based on patient selection criteria and response to previous therapies, often reflected in clinical guidelines. Assessment of the position in the treatment pathway drives the definition of the treatment population and choice of appropriate comparator. The pathway may have changed during the course of the trial if a new medicine has been approved or new clinical guidelines have been implemented locally. For example in cancer a new medicine may have been approved and reimbursed for patients with a specific genetic marker, thereby creating new sub-populations where different comparators are relevant. In some cases, the introduction of the medicine of interest will itself alter the treatment pathway. For some diseases, especially rare diseases, there may not yet be an established treatment pathway or sequencing, and as a result there may be variations across countries or health centres within countries.
2. Trial settings and sites do not reflect usual clinical practice
In multi-national studies the general level of care (concomitant therapies, access to technologies, patent support programmes) received by trial participants in some healthcare systems or study sites may differ from usual practice in the healthcare system of interest. This may have implications for the generalisability of the effectiveness results to that system. Recruitment of study participants may require a specific diagnostic activity that is currently not part of usual practice. Clinicians’ or participants’ willingness to participate or complete the trial may not be independent of factors such as patient adherence or underlying risk, which may be associated with effectiveness. This may be important when considering the applicability of results from studies in highly resourced healthcare systems such as US, where patients (in study centres) have increased access to sophisticated diagnostic and monitoring services and high-intensity care, to local populations where such services may not be available. If the effect of the medicine itself cannot be isolated from trial setting, then the combination of setting and intervention may need to be considered more broadly as an intervention strategy in its own right.
3. Other study design choices may limit generalisability
Designers of studies to demonstrate the safety and efficacy of a new intervention may need to address a number of (other) challenges. In heavily researched disease areas such as cancer, the recruitment of sufficient numbers of study participants, many of whom may be eligible for other trials, may be difficult. Time and cost pressures may result in shorter clinical studies measuring outcomes over relatively short time periods, requiring modelling of longer-term effectiveness endpoints to be undertaken. Such designs may result in more uncertainty for estimates of effectiveness, especially for interventions for chronic conditions or those with seasonal variations.