Context
A key objective of Work Package 1 of GetReal was to develop a framework for incorporating real-world data (RWD) into decision-making. Case studies were constructed to explore the different ways that RWD may be used to help demonstrate the relative effectiveness of new medicines.
Pragmatic clinical trials (PCTs) are randomised controlled trials (RCTs) that allow comparison of health interventions in diverse patient populations in routine clinical practice. Few PCTs have been carried out before marketing authorisation specifically to generate real-world data (RWD) on relative effectiveness to support regulatory assessment and health technology assessment (HTA). The purpose of this case study was to understand stakeholder views about PCTs conducted prior to market authorisation (‘early’ PCTs). It aimed to explore the timing and design of studies, the strength of evidence on effectiveness, acceptability for decision-making and the generalisability of results.
The case study focused on therapies for chronic obstructive pulmonary disease (COPD). Challenges to demonstrating the relative effectiveness of new medicines in COPD were identified, based on a review of publicly available European public assessment reports (EPARs) and HTA documents.
The Salford Lung Study was used as an example of an early PCT for a medicine carried out prior to marketing authorisation. It evaluated a new maintenance therapy for 2,800 COPD patients in 80 GP practices in North West England, using existing data collection systems and broad patient inclusion criteria. In particular, the study was able to include patients less likely to adhere to standard maintenance therapy.
The case study was also able to draw on the joint scientific advice offered by NICE and the MHRA prior to the Salford Lung Study.
A case study workshop was held to obtain views from GetReal participants and key stakeholders on the acceptability and usefulness of early PCTs.
Although COPD therapies and the Salford Lung Study provided the background for the discussion of early PCTs, the discussion is applicable to the wider use of ‘early’ PCTs.
What was examined in this case study?
The case study assessed the use of early PCTs to support estimates of relative effectiveness of new medicines. At the workshop three key questions were discussed:
- Question 1: Which effectiveness questions would pragmatic study designs best address?
- Question 2: How strongly would results from pragmatic studies be accepted as evidence?
- Question 3: How can the value and acceptability of PCTs be maximised?
What were the findings and conclusions?
The review of publicly available assessments of COPD medicines by HTA agencies in Europe (NICE, SMC, HAS and G-BA/IQWiG), Australia (PBAC), Canada (CADTH), undertaken by Context Matters, identified the following key themes relating to uncertainty in the clinical effectiveness evidence.
| Theme | Detail |
| Lack of appropriate end points | Omission of required end points such as morbidity and mortality. Agencies are interested in patient-centred outcomes (e.g. activities of daily living, health-related quality of life [HRQoL], exacerbations and symptoms) not the main trial focus |
| Clinically significant vs. statistically significant results | Agencies noted that their statistical significance in end points such as forced expiratory volume in one second (FEV1) and HRQoL did not translate into clinically meaningful results for patients |
| Study population issues | Exclusion of certain groups (e.g. with comorbidities or disease severity), resulted in study populations that were not comparable to clinical practice. |
| Short-term efficacy data | Relatively short duration of studies meant they were unable to capture important aspects of a chronic condition such as COPD |
| Inappropriate comparators | Concern about the lack of an active comparator (most clinical trials used placebo) |
| Lack of evidence for treatment pathway or combinations | Clinical data often did not include evidence of efficacy specific to a country’s treatment pathway, or the medicine’s use in combination with other therapies. |
| Weight of clinical uncertainty on the final outcome of HTA | Different remits, review processes and values result in areas of clinical uncertainty having different weights for HTA agencies (e.g. choice of an appropriate comparator is critical in Germany; varying acceptance of indirect comparison evidence in the absence of head-to-head data) |
At the case study workshop the majority view was that PCTs have a strong role in situations when RCTs cannot answer the question of effectiveness, which is most likely to be the case when there is a large ‘efficacy-effectiveness gap’ (for a definition, see Clarify the Issues). The strength of PCTs generally relates to external validity, which must be balanced against the weaknesses of a lack of internal validity and some difficulties with analyses.
Different ‘effectiveness challenges’ may require different levels of pragmatism. By reviewing clinical development programmes pharmaceutical research and development (R&D) can assess how best to address effectiveness challenges, by increasing the pragmatic elements of traditional RCTs or including new PCTs.
Workshop participants called for best practice guidelines on the use of early PCT designs, and insights from further collaborative efforts, such as case studies and evidence syntheses. Use of joint regulatory/HTA scientific advice processes may help determine how pragmatic elements can be brought into traditional RCTs to improve estimates of effectiveness and reduce uncertainty in decision-making by stakeholders. Although it is unlikely that early PCTs can or should be implemented for many medicines in development, ‘explanatory’ RCT designs may not always be most appropriate for generating estimates of relative effectiveness.
What do stakeholders say?
Question 1. Which effectiveness questions would pragmatic study designs best address?
- PCTs have a clear role when efficacy (from RCTs) is not predicted to match effectiveness, because of patient characteristics, comorbidities, real-world patient behaviour (for example, adherence), differences between old and new drug profiles, or service provision in health systems.
- Understanding of the likely ‘efficacy-effectiveness gap’ informs decisions on implementing PCTs.
- PCTs could allow enrolment of a greater number of patients than traditional RCTs and would allow broadening of populations of interest.
- Pragmatic designs may be beneficial for establishing effectiveness in subgroups of the general population, especially those excluded from conventional RCTs. A priori identification of subgroups is strongly preferred to post hoc subgroup derivation and evaluation.
- However, subgroup analyses in PCTs may be difficult to conduct, for instance in situations where there is variation in usual care.
- PCTs can generate valuable evidence on acceptability for patients in real practice, and on the position of new treatments within current treatment paradigms.
Question 2. How strongly would results from pragmatic studies be accepted as evidence?
- Strengths generally relate to external validity inherent in PCTs, weaknesses to lack of internal validity and analysis difficulties (for example, subgroups, treatment changes and multiple comparators).
- Clear objectives must be agreed by relevant stakeholders, pre-specified in a reporting and analysis plan, to allow for studies to be properly powered for subgroup analyses.
- PCT evidence is more acceptable for medicines with known benefit/risk profiles, less acceptable for medicines with novel mechanisms of action (uncertain efficacy and safety profile).
- PCTs were seen as complementing RCTs, and may be used to justify the clinical relevance of new medicines to decision-makers.
- PCTs should not replace RCTs: decision makers still require evidence that treatment effects are consistent with those observed in standard RCTs.
- A carefully formulated clinical development plan, based on broad stakeholder perspectives, should address when it is beneficial to include a traditional RCT, or when more elements of the pragmatic continuum (possibly using PRECIS criteria) should be introduced.
- Inclusion of a broader patient population and more clinically relevant end points in a typical phase 3 RCT may be more timely and efficient than introducing into the development plan a more real-world approach, with less intense monitoring or other more pragmatic elements.
- The robustness of long-term PCTs was questioned, for example, if randomisation breaks down following treatment switching.
- Uncertainty regarding the most appropriate trial design or evidence synthesis can lead to difficulties in interpreting results, and therefore lessen acceptability to decision-makers.
Question 3. How can the value and acceptability of PCTs be maximised?
- Both regulators and HTA agencies are concerned that development programmes do not answer the efficacy and effectiveness questions appropriately. There needs to be dialogue on how estimates are supported by the (planned) evidence package: it may be unrealistic to require PCTs in addition to RCTs.
- Develop guidelines on (pragmatic) trial design, evidence synthesis and best practice, driven by academia rather than pharmaceutical R&D. Initial focus should be on overarching principles, not specific details. Multi-stakeholder case studies evaluating how different pragmatic design elements can reduce uncertainty in different scenarios are valuable.
- A stepwise approach was suggested for implementing PCTs. Start with (explanatory) RCTs, relax exclusion criteria, and consider which aspects of the trial need to be more pragmatic.
- Upskilling on methodology and evidence synthesis is needed in both pharmaceutical companies and the public sector.
- Simulations may be used to increase generalizability, but will not be acceptable by all payers. There needs to be sufficient RWD sources from which to draw data to inform simulations.
- A framework is needed to help determine whether an ‘efficacy-effectiveness gap’ is to be expected. This in turn will determine whether more pragmatism is required in trials.
- Patients have a clear role in providing the ‘authentic voice’ required to make PCTs more accepted.
- More exploration of innovative trial designs is proposed. For instance, a hybrid PCT with an ‘RCT population’ could deliver improved internal validity for a study.
- Characterising options for ‘limited PCTs’ (i.e. pragmatic only to a certain extent), would help pharmaceutical R&D and decision makers consider which pragmatic elements to include in a PCT.
More details from the stakeholder workshop are found on the GetReal site (D1.5: Initial scope and D1.6: stakeholder views).
Key contributors
Mike Chambers, GSK
Páll Jónsson, NICE