Pajouheshnia R, Reitsma JB, Debray TPA
This study presents a new prediction model to guide treatment decisions in primary care patients at risk of CVD. We would like to compliment the authors for their extensive modeling strategies and detailed reporting. Although the model shows good calibration, we have concerns about its utility for assessing treatment benefit in individual patients. For a prognostic model to be used to guide treatment decisions, it should correctly estimate individual outcome probabilities under all treatment strategies being compared, in this case also the patient's untreated outcome probability if they were to remain untreated 1. This is challenging because, in real life, an individual's outcome cannot be observed under multiple treatment strategies. For this reason, researchers may choose to explicitly model predictors of treatment allocation and treatment-effect modifiers, or to restrict the analysis to a particular treatment exposure. Selak et al. have recognized this important issue and developed a prognostic model on individuals who remained untreated (with aspirin) by excluding baseline aspirin users and censoring post-baseline ("drop-in") users. Although this approach effectively removed treatment at baseline, selection bias is induced by simply censoring individuals treated during follow-up because aspirin treatment is likely initiated by the presence of certain risk factors. This has been shown to limit the generalizability of a prediction model 2. Unfortunately, this bias cannot be evaluated in the present study because the validation data is similarly affected.
Informative censoring is a well-known problem in non-randomized intervention studies and is commonly addressed using inverse probability weighting (IPW) strategies 3. Briefly, these strategies weight the analyses (e.g. development of a Cox model) by fixed, or ideally, time-dependent weights estimated using propensity score methods. In addition, alternatives to censoring patients to deal with treatment drop-in have been proposed. 4,5
Censoring due to aspirin drop-in may have affected the performance and generalizability of the developed model. We recommend the authors address potential selection bias by adopting IPW methods during development and validation. We further recommend that researchers who intend to develop a prognostic model 1) confirm whether the model is intended to guide treatment decisions, 2) if so, apply statistical methods to develop a model that correctly estimates "untreated risk", such as those outlined in 5, and 3) report information on treatment drop-in as recommended in 1. We hope this suggestion will increase the transparency of the often overlooked issue of treatment drop-in and promote better prognostic models to guide treatment decisions.
References Pajouheshnia, R., J. A. Damen, R. H. Groenwold, K. G. Moons and L. M. Peelen (2017). "Treatment use in prognostic model research: a systematic review of cardiovascular prognostic studies." Diagnostic and Prognostic Research 1(1): 15.
 Pajouheshnia, R., Peelen, L. M., Moons, K. G., Reitsma, J. B., and Groenwold, R. H. (2017). "Accounting for treatment use when validating a prognostic model: a simulation study." BMC medical research methodology, 17(1), 103.
 Hernán, M. A., S. Hernández-Díaz and J. M. Robins (2004). "A structural approach to selection bias." Epidemiology 15(5): 615-625.
 Groenwold, R. H., K. G. Moons, R. Pajouheshnia, D. G. Altman, G. S. Collins, T. P. Debray, J. B. Reitsma, R. D. Riley and L. M. Peelen (2016). "Explicit inclusion of treatment in prognostic modelling was recommended in observational and randomised settings." J Clin Epidemiol. 78:90-100
 Sperrin, M., G. Martin, A. Pate, T. Van Staa, N. Peek and I. Buchan (2018). "Using marginal structural models to adjust for treatment drop-in when developing clinical prediction models." Statistics in Medicine 37(28), 4142-4154.