Study design: Pragmatic trial

What is it?

Pragmatic trials aim to measure the relative effectiveness of treatment strategies in real-world clinical practice, as first described by Schwartz and Lellouch in 1967. They provide evidence of the added value of a treatment strategy in routine clinical practice, while maintaining the strength of a randomised controlled trial.

This entails the comparison of randomised groups of patients that are similar to the target group in the characteristics that modify drug response, in the setting where they would be treated in real life. The treatment strategies for comparison and outcome measures should be relevant for routine clinical practice. The term ‘pragmatic trial’ is commonly used for trials that asses the difference between treatment strategies, including extraneous factors (for example, the effects of co-medication, non-adherence and placebo effects), and aim to maximise generalisability to a broader setting or patient population.

For most new market-approved treatments, the clinical evidence (mainly phase 3 clinical trials) is still insufficient to fully guide clinicians and policy makers in choosing the optimal treatment for their patients. Pragmatic trials can help supplement this data with real-world evidence (RWE).

Pragmatic and explanatory trials: a continuum

Pragmatic and explanatory trials (which measure efficacy under ideal conditions, such as typical phase 3 RCTs) represent ends of a continuum rather than distinct entities (Thorpe et al 2009). A study may contain elements from both approaches.

The design choices that can be made towards a more pragmatic trial design relate to four domains: the study population; the setting of the trial; operationalisation of the intervention and choice of comparator treatment; and the outcome measure. General issues of data management and monitoring also need to be taken into account, because these can influence routine clinical practice and therefore the generalisability of the trial results.

A choice between a more pragmatic or more explanatory trial design can be made in the following examples:

  • Operationalisation of the intervention: extraneous effects (for example, compliance and co-medication) are either equalised in the comparison between the groups to study the true pharmacological effects of the medicine (more explanatory) or they are included in the full set of determinants of an overall treatment strategy (more pragmatic).
  • Outcome measure: either biologically meaningful outcomes are chosen (more explanatory) or meaningful outcomes for decision-making in routine clinical practice are chosen (more pragmatic).
  • Choice of participants: participants are either highly selected patients with a high probability to reveal a treatment effect (more explanatory) or the target population that is encountered in clinical practice (more pragmatic).

Importantly, a more pragmatic trial will aim to minimise the level of interference with clinical practice as far as possible, to maintain usual care throughout the trial.

Key elements of the continuum between pragmatism and explanation in trials are illustrated in the figure below.

Figure. Continuum between explanatory and pragmatic trials

pragmatic-explanatory-trials

In addition to the above choices in trial design leading to a more pragmatic trial, some specific trial applications may be considered for pragmatic trials. For example, if randomisation of patients in a medical setting changes the routine care process (for example, if introducing the intervention changes how care in general is provided at the site), cluster randomisation may be preferred (see also cluster RCTs). Another way to minimise the level of interference with clinical practice might be through a ‘cohort multiple RCT’ (cmRCT) design. Both options come with their own advantages and disadvantages.

Why is it useful?

  • Randomised study design: randomisation ensures that the comparison of benefit and risk between treatment groups is not confounded by incomparability of prognosis at baseline (‘prognostic incomparability’).
  • Trial results relevant to real-world populations: a more pragmatic trial enrols participants who are more similar to the target population than those in an explanatory trial (particularly for characteristics that may have an impact on the treatment response or ‘effect modifiers’, such as age or disease severity). They also use comparators and outcome measures relevant to clinical practice and attempt not to change routine clinical practice during the trial. This should lead to trial results that are more generalisable to patients in routine clinical practice than results from a more explanatory trial.
  • More realistic reflection of the likely treatment effects: the treatment response in a pragmatic trial is the total difference between treatment strategies, including associated compliance, placebo and other extraneous effects. Compared with an explanatory trial, this could result in a more realistic reflection of the likely treatment effects in patients treated in routine care.
  • Earlier evidence of effectiveness: use of a pragmatic trial may enable evidence of effectiveness to be provided at an earlier stage in the medicine development process (possibly even pre-launch, before regulatory approval) compared with an explanatory trial which only provides evidence of efficacy.

When is it suitable?

  • To guide decision-making on prescribing in routine clinical practice: pragmatic trials are suitable when the results of a trial are meant to directly guide decision-making on prescribing in routine clinical practice.
  • To determine effectiveness once safety (and efficacy) is established: if safety (and efficacy) of treatment has been established in a phase 2/3 clinical trial (or cannot be assessed further due to acute unmet needs, for example, the search for a vaccination for a life-threatening and fast-spreading infection), comparison with usual care in routine clinical practice is a good next step. This is particularly important when a difference between the effect of the medicine in the phase 2/3 trials and the effect of the medicine in the real world is expected. This efficacy–effectiveness gap (see a definition here) may be due to a number of factors, such as a difference in population or use of the treatments being compared including extraneous factors.

What are the limitations?

  • Operational challenges: pragmatic trial designs may lead to different and unanticipated operational challenges compared with typical phase 3 (explanatory) trials. For example, to involve real-world prescribers of a medicine, a study may be moved from specialised trial centres, often involved in explanatory trials, to a medical department or primary care setting, which may not be fully equipped or dedicated to support a clinical trial.
  • Complex interplay between study design and operational challenges and the implications: designers of pragmatic trials need to be aware of the consequences of their study design choices and balance the implications of these choices on generalisability to routine practice, validity, precision, acceptability to stakeholders and operational feasibility.

How are they designed?

Defining the research question and developing the trial protocol accordingly are crucial to ensuring that the purpose of a trial is clear and that reporting is transparent.

The design support tool PragMagic captures the most relevant design choices, their implications and the operational challenges that may be encountered when executing the trial. This enables a more explicit and transparent trial design process.

For more information about specific study design challenges, see here.

What do stakeholders say?

Stakeholder views on the ethical challenges and implementation of pragmatic trials in medicines development were elicited as part of GetReal through in-depth interviews (see Kalkman et al 2016). Key stakeholders interviewed included academia and independent research institutions, the pharmaceutical industry, regulators, health technology assessment (HTA) agencies and patient organisations.

Stakeholder views on the use of pragmatic trials have also been elicited through a GetReal case study on chronic obstructive pulmonary disease (see here).

Key contributors

Mira Zuidgeest, University Medical Center Utrecht
Iris Goetz, Lilly
Rick Grobbee, University Medical Center Utrecht