Background Treatment switching (TS) occurs in randomized clinical trials when patients discontinue their randomly assigned treatment and start a new therapy. Although ethically and clinically justified, TS presents a difficult problem for statisticians trying to ascertain the causal effects of interventions, particularly when assessing long-term time-to-event endpoints such as overall survival (OS). The difficulty arises as TS may occur after randomization but before observing the variable of interest, e.g., a death occurring after TS when assessing OS. Following the spirit of ICH E9(R1) on estimands, the clinical question of interest should drive how TS is handled in the analysis. In practice, especially when seeking regulatory approval, long-term time-to-event endpoints are assessed using an intention-to-treat (ITT) approach comparing treatments as they were initially randomized, i.e., regardless occurrence of TS. Additional methodologies were developed to assess treatment effect (TE) in the hypothetical scenario in which TS would not have occurred. These methods allow the relative effect of the experimental treatment over the control to be isolated by removing (or reducing) the potential benefit of switching. These methods include the rank preserving structural failure time (RPSFT), the two-stage accelerated failure time (2-AFT) and the inverse probability of censoring weighting (IPCW) models. Focusing on a special case of TS hereby referred to as cross over (CO), in which patients were only allowed to switch from control to experimental treatment, this work aims to i) contextualize these analyses in terms of the clinical question of interest, to ii) clarify their interpretation and role in regulatory submissions and to iii) provide an example of their application using a case-study in cell therapy.

Case-study TRANSFORM (NCT03575351) is a global, phase 3 study comparing lisocabtagene maraleucel (liso-cel) versus standard of care (SOC) as second-line therapy for primary refractory or early relapsed large B-cell lymphoma patients considered eligible for autologous stem cell transplantation. A total of 184 patients were randomized 1:1 to either the liso-cel or the SOC arms. CO was allowed upon confirmation by investigators of pre-defined clinical criteria. OS was one of the key secondary endpoints and was analyzed using the ITT approach while the RPSFT, 2-AFT and IPCW models were presented as pre-specified supportive analyses.

Results When the TE was estimated ignoring TS (i.e., following the ITT approach), the study did not demonstrate an improvement in OS (HR = 0.724; 95% CI: 0.443, 1.183). However, when the TE was estimated assuming CO did not occur, results from the 2-AFT and RPSFT models showed a favourable TE for liso-cel (HR = 0.415; 95% CI: 0.251, 0.686 and HR = 0.279; 95% CI: 0.145, 0.537, respectively). IPCW could not be implemented due to data limitations.

Discussion The definition of clear clinical objectives when evaluating long-term time-to-event endpoints is paramount to decide how TS should be analytically addressed. While regulators may be more inclined to focus on the ITT principle, other agencies such as payers may be also interested in evaluating the relative effect assuming TS did not occur. Together, these approaches contribute to a comprehensive evaluation of efficacy.

Event Free Survival (EFS) is considered a standard endpoint in Diffuse Large B-Cell Lymphoma (DLBCL). It measures the time to a composite event including tumor relapse, death or treatment failure, whichever occurs first. The first two types of events are traditionally well defined and not controversial in their ability to capture a detrimental outcome for the patient. On the other hand, treatment failure appears defined in very diverse ways across published trials, usually based on either the failure to reach complete or partial tumor response (CR/PR) by a given milestone, and/or on the administration of a further anticancer therapy. The contribution of this type of event to an objective assessment of treatment effect, comparatively to the two other types, appears harder to understand.

The review of a typical patient journey as referred to in the treatment guidelines helps to clarify the original clinical rationale for the definition of treatment failure in EFS. Achieving CR or PR has historically been a critical pre-requisite on the path to possible cure, in particular to enable the administration of high dose chemotherapy and stem cell transplantation (HDC/SCT).

However, the high diversity of implementation of the EFS endpoint across clinical studies, particularly in the definition of treatment failure, often blurs the relation to this original clinical rationale, and often doesn’t come with an explicit clinical explanation. This is even more challenging when the EFS endpoint is used in new drug development settings, e.g., with novel treatment approaches not relying on HDC/HCT. When different treatment modalities need to be compared, the definition of treatment failure may raise further difficulties in the statistical analysis and interpretation of study results.

In this presentation, we review possible patient journeys and trial situations to highlight how various definitions of EFS may address different questions of interest. This highlights how the definition of this endpoint would benefit from a structured and transparent estimand discussion according to ICH E9(R1), centered around explicit clinical assumptions and intercurrent events grounded into patient journeys. This is essential to identify the treatment effect(s) of interest and guide the trial design and statistical analysis, to ensure interpretability of the trial results. This presentation also highlights the need for statisticians to engage early on with clinical partners, emphasizing the importance of understanding the clinical rationale for endpoint conventions through the estimand framework.

Introduction/Background: We aimed to develop and evaluate data-generating models to jointly simulate outcomes and intercurrent events for randomised clinical trials to enable the assessment of properties of estimands.

Methods: We propose four data-generating models for the joint distribution of longitudinal continuous clinical outcomes and intercurrent events under the scenario where they are observable: a selection model, a pattern-mixture mixed model, a shared-parameter model and a joint model of longitudinally observed clinical outcomes and a survival model for intercurrent events. We present a case study in a short-term depression trial with repeated measurements of continuous outcomes and two types of intercurrent events, and compare the four proposed data-generating models.

Results: In our case study, we found that all four data-generating models can simulate different types of intercurrent events, their timing, and their associated longitudinal outcomes. These can be used to match envisaged patterns of intercurrent events and outcomes informed by prior available clinical trial data. For a given intercurrent event, the Shared-Parameter and Joint Models tend to associate more similar longitudinal profiles (because of shared latent random effects), while the Selection Model and Pattern-Mixture Model could allow more variation in associated profiles.

Conclusion: All four proposed data-generating models can be used to evaluate different estimands and to investigate their properties in-depth in the design stage. Thereby they are useful tools for the selection of estimands a priori.

While the industry and regulators’ interest in decentralized methods is long-standing, the Covid-19 pandemic accelerated and broadened the adoption and the experience with these methods in clinical trials. The key idea in decentralization is bringing the clinical trial design, typically on-site, closer to the patient’s experience (on-site or off-site). Thus, potential benefits of decentralizing studies include reducing the burden of participation in trials, broadening access of clinical trials to a more diverse population, or using innovative endpoints collected off-site.

This presentation facilitates evaluation of the added value and the implications of decentralized designs beyond the operational aspects of their implementation. The proposed approach is to use the ICH E9(R1) estimand framework to guide the strategic decisions around each decentralization component. Furthermore, the framework can guide clinical trialists to systematically consider the implications of decentralization, in turn, for each attribute of the estimand. Illustration of the use of this approach with a decentralized trial case study will show that the proposed systematic process can uncover the scientific opportunities, assumptions and potential risks associated with a possible use of decentralized components in the design of a trial. This process can also highlight the benefits of specifying estimand attributes in a granular way. The presentation will hence demonstrate that bringing a decentralization component into the design not only impacts estimators and estimation but can also correspond to addressing more granular questions, thereby uncovering new target estimands.

The goal of evaluation studies in supervised machine learning (ML) is the quantification of the generalization capability of a trained predictive model or a learning algorithm. For an unbiased estimation, data splitting is mandatory as the apparent training performance is not indicative of the targeted out-of-sample performance [1]. Various designs and methods have been proposed for a valid and efficient estimation, e.g. different cross-validation variations (k-fold, leave-one-out, grouped, stratified, nested). For ML practitioners, it is however often unclear which of the available approaches is most suitable for their specific problem.

In this work, we connect this issue to the ongoing estimand discussion in statistics [2]. We performed a selective literature review to summarize common solutions and pitfalls in the context of clinical risk prediction modelling. We derived a new framework that can guide ML practitioners through their estimand definition. We also conducted a range of numerical experiments with real and simulated data to investigate the consequences of inconsistencies between the target estimand and the actual experimental design. Moreover, we developed a R new package to allow the direct implementation of our framework in practice.

Our derived estimand framework requires the characterization of the theoretical estimand (“What is the estimation target?”) and the empirical estimand (“What is actually being estimated?”) [3]. For this purpose, the relevant patient population(s) and differences between development and implementation context are described by a set of constraints (e.g. “model implemented in the same clinic(s) where the development data has been sampled” vs. “model implemented in new clinic(s)”). Deviations between the theoretical and empirical estimand, which cannot be avoided in practice, need to be rectified by (transferability) assumptions. This enables a precise description of the relevant estimand(s) and the limitations of the evaluation study (unrealistic transferability assumptions). The framework allows to improve the current practice in the clinical risk prediction literature where the estimand is often only vaguely defined and the validity and usefulness of the performance estimate(s) is thus unclear. Our numerical experiments indicate that inconsistencies between theoretical and empirical estimand can lead to a severely biased performance estimation. To mitigate this issue in future ML evaluation studies, our new R package ‘mldesign’ provides a simple interface to transfer an estimand definition into a concrete study design (data splitting approach).

References:

1. Steyerberg, E. W., & Harrell, F. E. (2016). Prediction models need appropriate internal, internal–external, and external validation. Journal of clinical epidemiology, 69, 245-247.
2. Alpers, R. and Westphal, M. (2023). An estimand framework to guide model and algorithm evaluation in predictive modelling. Manuscript in preparation.
3. Lundberg, I., Johnson, R., & Stewart, B. M. (2021). What is your estimand? Defining the target quantity connects statistical evidence to theory. American Sociological Review, 86(3), 532-565.

*Both authors contributed equally and are listed in alphabetical order.