In practice, an efficacy-effectiveness gap (for a definition, see here) is likely to occur if a key characteristic (such as a patient-related or disease-related characteristic) is distributed differently in randomised controlled trials (RCTs) compared with real life.
What was examined in this case study?
This case study used a systematic literature review to:
- identify any efficacy-effectiveness gap between the published RCTs and observational studies on glucose-lowering medicines for patients with diabetes
- explore the drivers of effectiveness (factors that could influence this gap – for a definition, see here) in the evidence.
The review aimed to identify studies comparing glucagon-like peptide-1 analogues (GLP-1) with insulin and studies comparing dipeptidyl peptidase-4 inhibitors (DPP-4i) with sulfonylurea, and considering change in glycated haemoglobin (HbA1c) as an outcome. Relative effect estimates and characteristics of the study populations (age, sex, BMI, time since diagnosis and HbA1c) were compared across study designs.
What were the findings and conclusions?
In total the following numbers of studies were included:
- 11 RCTs (13 papers) and 7 observational studies (6 papers) comparing GLP-1 with insulin
- 16 RCTs (19 papers) and 4 observational studies (4 papers) comparing DPP-4i with sulfonylurea.
No efficacy effectiveness gap or drivers of effectiveness were identified: no clear differences in effect estimates across study designs were observed. Therefore, no evidence of an efficacy-effectiveness gap was identified. Also, no clear differences in patient characteristics across study designs were observed, meaning no potential drivers of effectiveness were identified.
However, based on the available studies, it is not possible to fully rule out the existence of an efficacy-effectiveness gap.
What are the limitations?
There are two possible reasons why an efficacy-effectiveness gap was not detected by the literature review:
- The observational studies may have been designed to mimic the RCTs. However this was not explicitly stated nor could it be deduced from inclusion criteria.
- Quality of the observational studies was poor (for example, no control for confounding) so the study effects may not reflect the actual effectiveness of the medicines.
The drivers of effectiveness investigated were limited to information available in both RCTs and observational studies (age, sex, BMI, time since diagnosis, baseline HbA1c). Therefore, other potential drivers of effectiveness (for example exercise and diet, comorbidity, co-medication, delivery of and adherence to treatment) could not be investigated.
Future studies based on patient-level data could investigate other potential drivers of effectiveness based on analyses of effect-modification.
The literature review compared relative effect estimates across study designs, by identifying studies where glucose-lowering medicines were compared with each other. An alternative is to compare the absolute effect of a specific medicine across study design. This would include studies not identified in the present review (e.g. the active treatment arm of a placebo RCT, and the relevant treatment arm of studies with a comparator not in the scope of this review).
Mikkel Z Ankarfeldt, Novo Nordisk