Effectiveness issues – Study design / setting

1. Trial treatment pathway is not generalisable to usual practice

The ‘treatment pathway’ refers to the sequence of previous treatments received, based on patient selection criteria and response to previous therapies, often as stipulated in clinical guidelines. Assessment of the new medicine’s position in the treatment pathway drives the definition of the treatment population and choice of appropriate comparator. This pathway may have changed during the course of the trial if a new medicine has been approved or a new clinical guideline has been implemented locally. In some situations, the introduction of the medicine of interest will itself alter the treatment pathway. For some new medicines there may not yet be an established treatment pathway, with variations in patient management between countries or between different health centres in the same country.

2. Trial settings and sites do not reflect usual clinical practice

In multi-national studies the level of supporting care (concomitant therapies, access to technologies, patent support programmes) received by trial participants in participating study sites in some healthcare systems may differ from the supporting care likely to be available in the healthcare system of interest. Recruitment of study participants may require diagnostic tests that are currently not part of usual practice. Participants’ willingness to participate in or complete the trial may not be independent of underlying risk factors, which may be associated with effectiveness. This may be important when considering the applicability of results from studies in highly resourced healthcare systems such as US, with increased access to sophisticated diagnostic and monitoring services as well as high-intensity care, to local populations where such services may not be available. If the effect of the medicine itself cannot be isolated from trial setting, then the combination of setting and intervention may need to be considered more broadly as an intervention strategy in its own right.

3. Other study design choices may limit generalisability

Designers of studies intended to demonstrate the safety and efficacy of a new intervention may need to address other challenges. In heavily researched disease areas such as cancer, the recruitment of sufficient numbers of study participants, many of whom may be eligible for other trials, may be difficult. Time and cost pressures may result in the implementation of clinical studies measuring outcomes over relatively short time periods, requiring modelling of longer-term effectiveness endpoints to be undertaken. Such designs may result in more uncertainty in resulting estimates of effectiveness, especially in the case of interventions for chronic conditions, or interventions where there are seasonal variations in incidence or outcome. Time and cost pressures may also encourage trial planners to locate study sites in healthcare systems where there is less historic trial involvement or where costs are less: generalisability of results from such sites to the healthcare system of interest may be questioned.