Background: Various treatments are recommended as first-line options in practice guidelines for depression, but it is unclear which is most efficacious for a given person. Accurate individualized predictions of relative treatment effects are needed to optimize treatment recommendations for depression and reduce this disorder's vast personal and societal costs.

Aims: We describe the protocol for a systematic review and individual participant data (IPD) network meta-analysis (NMA) to inform personalized treatment selection among five major empirically-supported depression treatments.

Method: We will use the METASPY database to identify randomized clinical trials that compare two or more of five treatments for adult depression: antidepressant medication, cognitive therapy, behavioral activation, interpersonal psychotherapy, and psychodynamic therapy. We will request IPD from identified studies. We will conduct an IPD-NMA and develop a multivariable prediction model that estimates individualized relative treatment effects from demographic, clinical, and psychological participant characteristics. Depressive symptom level at treatment completion will constitute the primary outcome. We will evaluate this model using a range of measures for discrimination and calibration, and examine its potential generalizability using internal-external cross-validation.

Conclusions: We describe a state-of-the-art method to predict personalized treatment effects based on IPD from multiple trials. The resulting prediction model will need prospective evaluation in mental health care for its potential to inform shared decision-making. This study will result in a unique database of IPD from randomized clinical trials around the world covering five widely used depression treatments, available for future research.

The estimand framework, introduced in the ICH E9 (R1) Addendum, provides a structured approach for defining precise research questions in randomised clinical trials. It suggests five strategies for addressing intercurrent events (ICE). This case study examines the principal stratum strategy, highlighting its potential for estimating causal treatment effects in specific subpopulations and the challenges involved. The occurrence of anti-drug antibodies (ADAs) and their potential clinical impact are important factors in evaluating biosimilars. Typically, analyses focus on subgroups of patients who develop ADAs during the study. However, conducting subgroup analyses based on post-randomisation variables, such as immunogenicity, can introduce substantial bias into treatment effect estimates and is therefore methodologically not optimal. The principal stratum strategy provides a statistical pathway for estimating treatment effects in subpopulations that cannot be anticipated at baseline. By leveraging counterfactuals to assess treatment outcomes, with and without the incidence of intercurrent events (ICEs), this approach can be implemented through a missing data perspective. We demonstrate the implementation of the principal stratum strategy in a phase 3 equivalence trial of a biosimilar for the treatment of rheumatoid arthritis. Using a multiple imputation approach, we leverage longitudinal measurements to create analysis datasets for subpopulations who develop ADAs as ICE. Our results highlight the principal stratum strategy's potential and challenges, emphasising its reliance on unobserved ICE states and the need for complex and rigorous modelling. This study contributes to a nuanced understanding and practical implementation of the principal stratum strategy within the ICH E9 (R1) framework.

Various statistical and machine learning algorithms can be used to predict treatment effects at the patient level using data from randomized clinical trials (RCTs). Such predictions can facilitate individualized treatment decisions. Recently, a range of methods and metrics were developed for assessing the accuracy of such predictions. Here, we extend these methods, focusing on the case of survival (time-to-event) outcomes. We start by providing alternative definitions of the participant-level treatment benefit; subsequently, we summarize existing and propose new measures for assessing the performance of models estimating participant-level treatment benefits. We explore metrics assessing discrimination and calibration for benefit and decision accuracy. These measures can be used to assess the performance of statistical as well as machine learning models and can be useful during model development (i.e., for model selection or for internal validation) or when testing a model in new settings (i.e., in an external validation). We illustrate methods using simulated data and real data from the OPERAM trial, an RCT in multimorbid older people, which randomized participants to either standard care or a pharmacotherapy optimization intervention. We provide R codes for implementing all models and measures.

Aims: Clinical guidelines often recommend treating individuals based on their cardiovascular risk. We revisit this paradigm and quantify the efficacy of three treatment strategies: (i) overall prescription, i.e. treatment to all individuals sharing the eligibility criteria of a trial; (ii) risk-stratified prescription, i.e. treatment only to those at an elevated outcome risk; and (iii) prescription based on predicted treatment responsiveness.

Methods and results: We reanalysed the PROSPER randomized controlled trial, which included individuals aged 70–82 years with a history of, or risk factors for, vascular diseases. We conducted the derivation and internal–external validation of a model predicting treatment responsiveness. We compared with placebo (n = 2913): (i) pravastatin (n = 2891); (ii) pravastatin in the presence of previous vascular diseases and placebo in the absence thereof (n = 2925); and (iii) pravastatin in the presence of a favourable prediction of treatment response and placebo in the absence thereof (n = 2890). We found an absolute difference in primary outcome events composed of coronary death, non-fatal myocardial infarction, and fatal or non-fatal stroke, per 10 000 person-years equal to: −78 events (95% CI, −144 to −12) when prescribing pravastatin to all participants; −66 events (95% CI, −114 to −18) when treating only individuals with an elevated vascular risk; and −103 events (95% CI, −162 to −44) when restricting pravastatin to individuals with a favourable prediction of treatment response.

Conclusion: Pravastatin prescription based on predicted responsiveness may have an encouraging potential for cardiovascular prevention. Further external validation of our results and clinical experiments are needed.

Observational data provide invaluable real-world information in medicine, but certain methodological considerations are required to derive causal estimates. In this systematic review, we evaluated the methodology and reporting quality of individual-level patient data meta-analyses (IPD-MAs) conducted with non-randomized exposures, published in 2009, 2014, and 2019 that sought to estimate a causal relationship in medicine. We screened over 16,000 titles and abstracts, reviewed 45 full-text articles out of the 167 deemed potentially eligible, and included 29 into the analysis. Unfortunately, we found that causal methodologies were rarely implemented, and reporting was generally poor across studies. Specifically, only three of the 29 articles used quasi-experimental methods, and no study used G-methods to adjust for time-varying confounding. To address these issues, we propose stronger collaborations between physicians and methodologists to ensure that causal methodologies are properly implemented in IPD-MAs. In addition, we put forward a suggested checklist of reporting guidelines for IPD-MAs that utilize causal methods. This checklist could improve reporting thereby potentially enhancing the quality and trustworthiness of IPD-MAs, which can be considered one of the most valuable sources of evidence for health policy.

Predicting future outcomes of patients is essential to clinical practice, with many prediction models published each year. Empirical evidence suggests that published studies often have severe methodological limitations, which undermine their usefulness. This article presents a step-by-step guide to help researchers develop and evaluate a clinical prediction model. The guide covers best practices in defining the aim and users, selecting data sources, addressing missing data, exploring alternative modelling options, and assessing model performance. The steps are illustrated using an example from relapsing-remitting multiple sclerosis. Comprehensive R code is also provided.

The use of disease or multi-use models (i.e. one model that can be used to determine the cost effectiveness of many new technologies in a certain disease) is increasingly being advocated. Diabetes is a disease area where such models are relatively common; examples are the CORE Diabetes Model, which was used for National Institute for Health and Care Excellence (NICE) Guideline NG28, and the UK Prospective Diabetes Study (UKPDS) model. Recently, Eli Lilly has submitted a new diabetes model, the PRIME Type 2 Diabetes Model (PRIME T2D), to the NICE. This model was used in the NICE single technology appraisal (STA) process of tirzepatide (tradename Mounjaro®; Technology Appraisal 924). In this commentary we set out the key learnings from the External Assessment Group (EAG) perspective regarding this TA, the assessment of a disease model, and associated challenges.

Aim: Comparative effectiveness research using real-world data often involves pairwise propensity score matching to adjust for confounding bias. We show that corresponding treatment effect estimates may have limited external validity, and propose two visualization tools to clarify the target estimand.

Materials & methods: We conduct a simulation study to demonstrate, with bivariate ellipses and joy plots, that differences in covariate distributions across treatment groups may affect the external validity of treatment effect estimates. We showcase how these visualization tools can facilitate the interpretation of target estimands in a case study comparing the effectiveness of teriflunomide (TERI), dimethyl fumarate (DMF) and natalizumab (NAT) on manual dexterity in patients with multiple sclerosis.

Results: In the simulation study, estimates of the treatment effect greatly differed depending on the target population. For example, when comparing treatment B with C, the estimated treatment effect (and respective standard error) varied from -0.27 (0.03) to -0.37 (0.04) in the type of patients initially receiving treatment B and C, respectively. Visualization of the matched samples revealed that covariate distributions vary for each comparison and cannot be used to target one common treatment effect for the three treatment comparisons. In the case study, the bivariate distribution of age and disease duration varied across the population of patients receiving TERI, DMF or NAT. Although results suggest that DMF and NAT improve manual dexterity at 1 year compared with TERI, the effectiveness of DMF versus NAT differs depending on which target estimand is used.

Conclusion: Visualization tools may help to clarify the target population in comparative effectiveness studies and resolve ambiguity about the interpretation of estimated treatment effects.

In recent years, there has been a growing interest in the prediction of individualized treatment effects. While there is a rapidly growing literature on the development of such models, there is little literature on the evaluation of their performance. In this paper, we aim to facilitate the validation of prediction models for individualized treatment effects. The estimands of interest are defined based on the potential outcomes framework, which facilitates a comparison of existing and novel measures. In particular, we examine existing measures of discrimination for benefit (variations of the c-for-benefit), and propose model-based extensions to the treatment effect setting for discrimination and calibration metrics that have a strong basis in outcome risk prediction. The main focus is on randomized trial data with binary endpoints and on models that provide individualized treatment effect predictions and potential outcome predictions. We use simulated data to provide insight into the characteristics of the examined discrimination and calibration statistics under consideration, and further illustrate all methods in a trial of acute ischemic stroke treatment. The results show that the proposed model-based statistics had the best characteristics in terms of bias and accuracy. While resampling methods adjusted for the optimism of performance estimates in the development data, they had a high variance across replications that limited their accuracy. Therefore, individualized treatment effect models are best validated in independent data. To aid implementation, a software implementation of the proposed methods was made available in R.