What is the issue being addressed by this case study?
One of the biggest limitations of the cohort multiple randomised controlled trial (cmRCT) study design is the rate of drop out in the treatment group. Participants in the treatment group are asked for consent after they are randomised. This risks unbalanced treatment groups (selection bias) if patients refuse treatment. If the patients who refuse treatment are still followed up, it may lead to a dilution of the treatment effect and a drop in the overall statistical power, which affects the ability of the trial to show any true differences in effects between the treatment and control.
GetReal examined drop out from cmRCT trials using a case study of patients with cardiovascular disease.
What was examined in this case study?
This case study aimed to see if the high drop-out rate from cluster cmRCT studies can be accounted for in the analysis of study results.
It used simulations of a hypothetical cluster cmRCT trial comparing a novel intervention with a control group of patients at risk of cardiovascular disease (CVD).
The simulations adjusted specific characteristics of the participants in the hypothetical cluster cmRCT trial, including the high rate of drop out, which is expected in the treatment group for cmRCT trials, and the correlation between this drop out and patient health.
Four different methods of accounting for bias caused by drop out were compared to determine which method produced results that were closest to the actual treatment effect:
- Intention-to-treat analysis: all participants are included in the analyses, even if they dropped out of the trial. In this method, there are usually assumptions made about the participants who dropped out of the trial, such as poor response to the treatment if they remained in the trial.
- Per-protocol analysis: only participants who receive the treatment are included in the analyses.
- Two-stage predictor substitution (2SPS): predicted values for treatment received are used as an explanatory variable.
- Two-stage residual inclusion (2SRI): similar to the 2SPS but both treatment received and the residuals are used.
Both 2SPS and 2SRI account for unmeasured study characteristics that may have an effect on the treatment (or unmeasured confounding) and both are known as instrumental variable analysis. These methods exploit information about the relationship between the treatment assigned and the treatment received in order to calculate the relationship between treatment received and the outcome of interest for the participants that comply with treatment (complier average causal effect [CACE]).
What were the findings and conclusions?
- The cmRCT can be an efficient design for conducting pragmatic trials, however there are still many questions to answer.
- Interpretation of the treatment effect can be difficult given the timing of treatment refusal. The type of intervention and whether refusal is related to participation in the trial will affect which analysis method is more relevant.
- Even with the 2SRI method, bias in estimation of the CACE can still be present in many scenarios. Simulations should be run to mimic the trial of interest to see if the cmRCT design is a viable option.
- Large drop-out rates in cmRCT trials cannot solely be accounted for in the analysis; it should also be considered in the study design (for example, when determining the number of participants the study or sample size calculations).
What do stakeholders say?
Stakeholder feedback on the use of these methods has not been sought as part of GetReal. Instrumental variable methods are used widely in observational studies where confounding is often an issue. However, it is arguable that the use of this method in the setting of a randomised design is easier to verify than in an observational study so it is likely to be deemed acceptable for the cmRCT design by methodologists.
For further information, see Pate et al., 2016.
Key contributor
Alexander Pate, University of Manchester