Population: effectiveness vs. efficacy

Is the population of interest different to that in registration randomised controlled trials?

This page takes you through the difference between efficacy and relative effectiveness with respect to the population (Table 1 below) and reasons why they may be different. It then outlines possible planning questions which may help you to anticipate potential effectiveness issues that may arise when undergoing health technology assessment (HTA). The page concludes with links to a number of methods that can be used to further explore potential effectiveness issues for a medicine related to the population.

Table 1. Difference between efficacy and relative effectiveness with a focus on the population

Efficacy

The change in a key clinical measure compared to placebo in a cohort of well-defined patientsa under a standardised care protocol.

 Relative effectiveness

The change in a variety of endpoints of interest to patients and providers compared to the usual care provided in the population of patients identified as eligible for treatment by clinicians under the conditions for normal care in the healthcare system, subject to free and variable clinician and patient behaviour.
aThere are good reasons for the choice of population and definitions of inclusion and exclusion criteria in an RCT (following rules set by regulatory agencies): feasibility/time required to recruit and/or stratify, ethical considerations, patient willingness to enrol.

Why is the population of interest different?

Clinicians decide which treatment to offer, taking into account the clinical characteristics, prognosis, circumstances and preferences of the individual patient. They operate in a healthcare system which sets priorities and constraints on care, and they may be incentivised to treat certain patient groups. They may be required to follow (clinical) guidelines regarding the eligibility of patient groups for specific treatments. Clinicians will need to decide on the best treatment to offer to patients with a condition where there is a licence to administer the medicine but with certain characteristics (for example old age or co-medication) for which subjects were excluded from or under-represented in clinical trials.

Patients engage with clinicians on treatment choice, and patient preference influences the treatments used. Selection criteria for clinical trials may mean that certain patient populations are excluded. Patient adherence to a new treatment may be different in real-world practice compared to clinical trials, requiring adjustment to the relative effectiveness assessment.

Healthcare systems set priorities and develop clinical guidelines, for example based on health technology assessments (HTA) and expert opinion. There may be specific restrictions placed on patient eligibility for medicines, and reimbursement may only be available for sub-groups of patients with conditions that have a license to receive the medicine due to considerations of value or budget impact.

Potential assessment issues and related planning questions

The table below includes potential issues that may arise in the assessment of a medicine and questions that may help to understand these issues.

Table 2. Issues and planning questions related to the population

Issue Planning questions
The population in the RCT is too narrow/skewed (for example, with respect to age, gender, comorbidity) compared with the expected real-world population, and so the effect size is not necessarily generalisable •      How do population characteristics differ between the RCT and equivalent (local) real-world population?

•      Are any differences expected to result in a different effect size?
The population expected to be treated is at an earlier or later position in the care pathway than that included in the RCTs (for example, first-line therapy in the RCT, but in the real world treatment will be second or third line). So there are no data on effect size in population of interest.

The care pathway at launch of the medicine may not be the same as at the start of the RCT

The care pathway may vary by country

Treatment using the medicine may not be consistent with product labelling

 •      How are patients managed in the real world?

•      Where is the new medicine likely to be in the care pathway?

•      Are there any trial patients with the same prior treatment history as the expected real-world patients?

•      Will the care pathway change significantly by the time of launch? (to understand the care pathway see Understanding the Care Pathway or the Patient Journey)
RCT participant selection criteria are applied to a different standard in real-world practice (for example, clinician opinion may not be confirmed by all available laboratory or other tests). So the population treated may differ from population tested

This may be exacerbated by the use of new genetic or biological screening

Concern that a larger number of patients may be treated in real-world practice, many of whom will not be able to benefit to the same degree

 •      How are patients diagnosed and characterised in real-world practice?

•      How does this differ to RCT criteria?

•      Can target patients be identified using existing information in real-world clinical practice?

•      Will target patients be identifiable in the real world without a change in clinical practice or investment in new tests/technology?
The RCTs do not include ‘local-type’ patients, so trial efficacy is not generalisable to local practice

What may seem to be ethnic or sociodemographic differences may be due to differences in the care pathway, or in the availability of comparators and other interventions

 •      Are there geographical or ethnic differences that are expected to influence effect size? Do RCTs include sufficient patients to generalise across these groups?

•      Are there protocol instructions such as ‘best supportive care’ that may vary significantly by geography?

•      Do the RCT populations meet the known requirements of HTA bodies?

Methods to explore potential issues

To explore how the choice of study population may impact a medicine’s effect estimate, there are a number of methods you may use. See Methods to Explore and Identify Drivers of Effectiveness.

In addition, it may be useful to examine if channeling bias has occurred for previously authorised medicines in the disease area of interest and to consider if this may be an issue for the medicine of interest. Channelling bias occurs when a newly marketed medicine and an established medicine with similar therapeutic indications are prescribed to different patients according to their prognostic characteristics. This was explored in GetReal case studies in 3 disease areas: anti-coagulant medicines, anti-hypertensive medicines, and anti-diabetic medicines. These can be found at the following RWE Navigator pages below:

It may also be helpful to consider reviewing existing health technology assessments in the disease area: see Existing Health Technology Assessments

 

To start to find potential options using RWE to address the challenges you have identified, see

OR