Pragmatic trial: study design considerations

A pragmatic trial aims to capture the true effect of a treatment strategy in the real world. Selecting the setting, study population, mode of intervention, comparator and outcome are crucial in designing pragmatic trials. In combination with monitoring and data collection that does not change routine care, this will enable appropriate generalisation to the target patient group in clinical practice. To benefit from the full potential of pragmatic trials, there is a need for guidance and tools in designing these studies while ensuring operational feasibility.

PragMagic is a decision support tool for pragmatic trial design. As the generalisability of trial results to routine clinical practice is considered a key aspect of a pragmatic trial, generalisability plays a central role in this tool. The PragMagic tool aims to support trial design teams to maximise generalisability of trial findings to the routine care setting of interest while ensuring validity and operational feasibility of the trial. PragMagic gives insight into the possible consequences of more pragmatic trial design choices and operational challenges on generalisability, validity, precision, stakeholder & ethical acceptability and operational feasibility

Below the different study design aspects on which choices have to be made are described with links to further discussions on these design choices and related challenges and implications.

Selection and inclusion of usual care sites

The sites studied in a trial, with their specific expertise, care pattern and population, may influence the observed treatment effect and generalisability of the study results to the target population. Actual participation of sites and subsequent patient enrolment also determine successful execution of a trial. A balance is needed between studying representative sites, regarding factors that could modify the treatment effect, and conducting the trial efficiently.

To learn more about possible modifiers of treatment effect (or ‘effect modifier’) related to sites and selecting sites for pragmatic trials (assessment of feasibility and capacity, training and supporting sites) see here.

Participant selection, recruitment and retention

The population studied in a pragmatic trial can influence the observed treatment effect in the trial. For example, if drug metabolism changes with age, the observed effects or side-effects could be different when studying a population including children or elderly people. Pragmatic trials aim to include all patients eligible to receive the experimental treatment in real life. However, the population studied is not only determined by the eligibility criteria, some selection will occur when inviting patients for the trial. This may reduce the generalisability of the results to the target population and the efficiency of running the trial.

A balance is needed between including a representative population for modifiers of treatment effect (or ‘effect modifier’) in the target population and the feasibility of running the trial. To learn more about how participant selection influences generalisability, and which factors affect participant recruitment and retention in pragmatic trials see here.

Informed consent

A pragmatic trial aims to study the effect of a drug under real-world conditions. However, concerns have been raised that current ethical regulations compromise the real-world nature of a pragmatic trial. To address the challenges of traditional informed consent and explore the possibilities of alternative consent models, the following assumptions were made:

  • Not all requirements for regulatory informed consent for randomised trials are always necessary to protect the rights and interest of participants in pragmatic trials.
  • The requirements for regulatory consent impede the scientific and societal goals of pragmatic trials.

To learn more about the challenges of informed consent, proposed alternative consent models and their implications see here.

Defining questions, choosing comparators, allocating treatments

To adequately guide clinicians’ treatment decisions, pragmatic trials should deliver evidence on the added value of new medication compared to usual care for those patients eligible to receive the new treatment in day-to-day clinical practice. However, variations in usual care and suboptimal care may complicate the operationalisation of such a comparator arm. Treatment-related study procedures, including allocation and implementation of treatment strategies, should resemble clinical practice as closely as possible, and decisions on drug dosage, co-interventions, and the management of adverse effects should be left to the treating clinician. The specific objectives of the trial should be carefully considered because this determines patient eligibility, allowed treatment switches between arms and the suitability of a superiority or non-inferiority trial (used to test a hypothesis that the new treatment is superior or non-inferior to usual care).

To learn more about challenges regarding comparator choice, mode of intervention and treatment allocation for pragmatic trials see here.

Outcome selection and measurement

The selection of a primary outcome is a key design choice for any trial. The set of outcomes (primary and secondary) in pragmatic trials should be relevant for patients, clinicians and for decision-making regarding the value of treatment in practice. In contrast to explanatory trials, treatment effect estimates should include extraneous effects of co-medication, non-adherence and placebo effects as in clinical practice. For this reason patients and clinicians are usually not blinded (open-label) in pragmatic trials and outcome measurement should not interfere with usual clinical practice.

To learn more about the selection and definition of outcome measures in pragmatic trials, including surrogate- and patient-reported outcomes and how to measure them while satisfying stakeholders and ensuring valid and precise estimates of treatment effect see here.

Collecting and reporting safety data and monitoring trial conduct

Pragmatic trials fall under the interventional trial category of the European Clinical Trial Directive. This results in obligations for monitoring of safety and trial conduct that may affect routine clinical care and have an impact on patient and clinician behaviour. Current international guidelines state that in an interventional trial all serious adverse events must be collected, and reported within 24 hours to the trial sponsor unless specified otherwise in the study protocol. The collection and reporting of non-serious adverse events may be tailored for the study and depends on the added risks of the trial intervention relative to standard care.

To learn more about possible approaches to ensure the timely capture of sufficiently detailed data on serious adverse events without influencing the generalisability of trial results and about proportionate and risk-based methodologies for monitoring of trial conduct see here.

Data collection

The collection of high-quality data is essential to any trial. Collection of such detailed data typically requires on-site staff training in data collection, and includes quality checking. These requirements can present a burden to study sites. They may prevent research-naive clinicians from participating in trials and considerably interfere with usual care and the real-world nature of the trial, thus limiting the generalisability of study results. Therefore, pragmatic trials should use existing routine systems (such as electronic health records [EHRs]) and processes as much as possible. Typically, pragmatic trials take a hybrid approach between the highly controlled, dedicated case report form (CRF)-based approach and using data routinely collected in, for example, EHRs.

To learn more about challenges with the use of existing routine systems, technical options for data collection, and international initiatives dedicated to improving the standardisation and quality of routinely collected data click here.

In addition to study design considerations, there may be issues with the analysis of pragmatic trials. To see simulation studies completed as part of GetReal that aimed to control for bias with pragmatic trial see here.