Aim: Comparative effectiveness research using real-world data often involves pairwise propensity score matching to adjust for confounding bias. We show that corresponding treatment effect estimates may have limited external validity, and propose two visualization tools to clarify the target estimand.

Materials & methods: We conduct a simulation study to demonstrate, with bivariate ellipses and joy plots, that differences in covariate distributions across treatment groups may affect the external validity of treatment effect estimates. We showcase how these visualization tools can facilitate the interpretation of target estimands in a case study comparing the effectiveness of teriflunomide (TERI), dimethyl fumarate (DMF) and natalizumab (NAT) on manual dexterity in patients with multiple sclerosis.

Results: In the simulation study, estimates of the treatment effect greatly differed depending on the target population. For example, when comparing treatment B with C, the estimated treatment effect (and respective standard error) varied from -0.27 (0.03) to -0.37 (0.04) in the type of patients initially receiving treatment B and C, respectively. Visualization of the matched samples revealed that covariate distributions vary for each comparison and cannot be used to target one common treatment effect for the three treatment comparisons. In the case study, the bivariate distribution of age and disease duration varied across the population of patients receiving TERI, DMF or NAT. Although results suggest that DMF and NAT improve manual dexterity at 1 year compared with TERI, the effectiveness of DMF versus NAT differs depending on which target estimand is used.

Conclusion: Visualization tools may help to clarify the target population in comparative effectiveness studies and resolve ambiguity about the interpretation of estimated treatment effects.

Background: Previous studies suggest that hemodiafiltration reduces mortality compared to hemodialysis in patients with end-stage kidney disease (ESKD), but controversy surrounding its benefits remain and it is unclear to what extent individual patients benefit from hemodiafiltration. This study aimed to develop and validate a treatment effect prediction model to determine which patients would benefit most from hemodiafiltration compared to hemodialysis in terms of all-cause mortality.

Methods: Individual participant data from four randomized controlled trials comparing hemodiafiltration with hemodialysis on mortality were used to derive a Royston-Parmar model for prediction of absolute treatment effect of hemodiafiltration based on pre-specified patient and disease characteristics. Validation of the model was performed using internal-external cross validation.

Results: The median predicted survival benefit was 44 (Q1-Q3: 44-46) days for every year of treatment with hemodiafiltration compared to hemodialysis. The median survival benefit with hemodiafiltration ranged from 2 to 48 months. Patients who benefited most from hemodiafiltration were younger, less likely to have diabetes or a cardiovascular history and had higher serum creatinine and albumin levels. Internal-external cross validation showed adequate discrimination and calibration.

Conclusion: Although overall mortality is reduced by hemodiafiltration compared to hemodialysis in ESKD patients, the absolute survival benefit can vary greatly between individuals. Our results indicate that the effects of hemodiafiltration on survival can be predicted using a combination of readily available patient and disease characteristics, which could guide shared decision-making.