Effectiveness issues: challenges in demonstrating effectiveness

What are effectiveness issues?

When evidence of (relative) effectiveness of a new medicine is presented to health technology assessment (HTA) agencies, a variety of issues or challenges to generating or using the evidence may be raised. These are referred to here as effectiveness issues.

Effectiveness issues may be classified using the PICOS acronym, which defines five categories of issues: population, intervention, comparator, outcomes, and study design or study setting. The PICO framework was formulated in the 1990s to frame and answer clinical or healthcare related questions in Evidence Based Medicine (Richardson et al., 1995).

Choose a PICOS category from the left-hand menu to find out more information about the different types of issues.

Understanding effectiveness issues

Sponsors of new medicines aim to understand likely effectiveness issues, through:

  • review of recent assessments of medicines with similar characteristics
  • interpretation of guidance for making submissions
  • direct consultation with experts and the agencies themselves (‘scientific advice’).

Using this information together with a thorough understanding of the new medicine, sponsors will design and execute programmes of studies and analyses that generate evidence of relative effectiveness that is likely to be acceptable to different decision makers. As a result, effectiveness issues will have been anticipated and prevented.

Effectiveness issues in decision-making

For decision-making at the time of marketing authorisation of a new medicine, the main focus is on the pivotal studies (Phase 2b and Phase 3 clinical trials) that are powered to detect superiority or non-inferiority in an efficacy outcome of importance. In addition to the results of these studies, evidence submitted to HTA agencies is likely to contain further information from evidence syntheses: systematic reviews, meta-analyses and indirect comparisons (which are especially valuable in the absence of head-to-head data), as well as from statistical or other predictive models and supporting observational studies.

HTA agencies consider the extent to which the evidence presented is relevant (‘generalisable’) to clinical practice and the population in their country who would be likely to be eligible to receive the new medicine. They firstly assess whether the evidence presented can form the basis of a decision, as well as the risks of not making any decision or of making an incorrect decision.

Agencies may place a different emphasis on particular aspects of the evidence presented, depending on their views of the strength of different types of evidence and what type of evidence is optimal to support their decision-making in the particular disease area. It is quite likely that a perfect package of evidence cannot be generated to meet the needs of all decision makers, given these differences and also the time and budget constraints for pivotal studies.


P. Population

Is the study population well-defined and relevant? Does it represent the population for whom the decision is being made?

Read More

I. Intervention

Is the study intervention (diagnosis, medicine, administration, monitoring) applicable in routine practice? In addition to the medicine itself, the intervention may include diagnostic work-up, the method administration, patient monitoring and termination of therapy (which may be determined by stopping rules).

Read More

C. Comparator

Does the study comparator represent current usual care, or ‘Standard of Care’, which would be replaced by the new intervention? (Occasionally the new intervention may be added to usual care, in which case the comparator is usual care and the intervention is usual care combined with the new intervention).

Read More

O. Outcome

Do the outcomes reported by the trial (or those predicted, based on the trial results) capture all relevant aspects of relative effectiveness of the new medicine? And are they sufficient to form the basis of a decision?

Read More

S. Study Design / Setting

Does the study design or study setting support the demonstration of effectiveness? This may include when the study was conducted, if there have been changes in managing patients with the condition of interest.

Read More