The estimand framework, outlined in the ICH-E9 addendum, can help research teams guide their choices among several design and analytical methods, to those that result in better aligned estimators to the main questions or estimands of interest in clinical development. Several recommendations and guidelines exist facilitating the use of this framework to refine efficacy questions or achieve better alignment of estimators to the estimand of interest for pivotal trials. In contrast, few such guidelines exist for refining or aligning with safety estimands whether in pivotal or post-market studies. While several safety methods and estimators exist, such as proportion of events or incidence of events in follow-up, what estimand they align to is often poorly understood in the presence of censoring, intercurrent events, or competing risk.

In this presentation, we demonstrate that the estimand framework is broadly applicable to safety questions, especially relating to those safety topics of special interest, across different phases of development. We will motivate the use of the estimand framework in safety in pivotal, and post-market observational studies. In all studies, we will discuss the impact of safety-specific concepts, such as time-at-risk, rare incidence, and risk mitigation, on the estimand thinking and associated analytical strategies. In observational studies, we will also advocate for the complementarity of the causal inference thinking (e.g., use of the target trial framework concept) and the estimand framework. Finally, we will review and critique existing safety estimators in light of these frameworks and concepts and highlight areas where further methodological work is needed.

Further investigation of the long-term benefit and risk profile of a new therapy after approval in a real-world setting is essential to ensure relevant information is made available to patients and health care providers for the safe use of a drug. Our goal is to improve the understanding of two analysis methods of post-marketing safety studies (1). The motivating case study is a post-marketing observational study for a biologic drug investigating potential long-term risk of malignancy. The main methodological challenge is to answer a comparative safety question, in the real-world setting, where subjects switch away or to the investigational drug during their follow-up. Standard approaches include an “ever/never exposed” type of design where the interpretation becomes challenging in the presence of frequent treatment switching. Further declinations of this approach have been proposed, in which time and events assignment to respective treatment arms differ. In the so called “experimental hierarchical approach”, the experimental arm is given a higher ranking and time and events occurring after initiation of the investigating drug are assigned to the investigating drug irrespective of further treatment change. In the “overlapping approach”, on the contrary, both arms are handled equally and given same importance. Both approaches attempt to evaluate a treatment effect accounting for possible treatment switch. We explain these two estimators with the help of multistate model methodology and investigate both approaches under the Nullhypothesis of no group difference. This leads to a discussion of the timescale and the rational is that the estimators are biased under the Nullhypothesis. Finally, we show a possible improvement to be considered. A simulation study will evaluate the type 1 error rate and power of the different analytical methods under different switching and incidence scenarios.

(1) Joel M Kremer, Clifton O Bingham III, Laura C Cappelli, Jeffrey D Greenberg, Ann M Madsen, Jamie Geier, Jose L Rivas, Alina M Onofrei, Christine J Barr, Dimitrios A Pappas, et al. Postapproval comparative safety study of tofacitinib and biological disease-modifying antirheumatic drugs: 5-year results from a united states–based rheumatoid arthritis registry. ACR open rheumatology, 3(3):173–184, 2021.

Background: 

Adverse events (AEs) in clinical trials (CT) are usually tabulated by frequency and severity. Such separate descriptive summaries cannot reflect the overall AE burden of the treatment and do not provide statistical inference on overall between-treatment differences. The aim of this project is to adopt an AE burden score (AEBS) – a versatile quantitative measure (Le-Rademacher et al. 2020) and to evaluate suitable statistical approaches for comparing treatment effect on AE burden.

Materials/Methods:  

For illustration, data from two randomized double-blind placebo controlled CTs (Trials 1 and 2) were explored. The main interest is on the AE burden in the first 12 weeks when the dose was uptitrated weekly to identify the maximum tolerated dose for individual patients. The experimental treatment was known to cause some specific AEs.

The AEBS is a continuous, exposure adjusted metric incorporating duration, severity, and frequency of AEs. For patients without AEs, the AEBS is zero. For some AE episodes information of severity or/and duration was incomplete, therefore AE rate (frequency divided by exposure duration) was also explored. AEBS and AE rates were separately derived for treatment-specific and all AEs.

The distribution of AEBS is skewed with excess zero-values, with mean and variance positively correlated. These features restrict choice of applicable statistical models. For AEBS we evaluated: 1) ANOVA model (reference model); 2) ANOVA model using the natural logarithm of the AEBS plus a small value (delta), i.e., ln(AEBS+delta) to allow including patients without AEs; 3) Tweedie regression; 4) quantile regression for median. For AE rates, 5) negative binomial regression model with exposure duration as offset was evaluated.

The ability of a model to discriminate AE burden (AEBS or AE rate) between treatment groups was assessed based on the ratio of treatment effect and its standard error. Where applicable (models 1-3), fit of the models was compared using the Akaike criterion (AIC).

Results/Discussion:  

Trial 1: For models 1-5 the estimated discrimination abilities for all/treatment-specific AEs are: 6.8/14.2; 10.5/16.3; 9.3/17.1; 8.2/14.9; 7.4/16.4. Treatment separation is clearly better for treatment-specific AEs. For AEBS, Tweedie regression is the best for treatment-specific AEs and second-best for all AEs.

Trial 2: For models 1-5 the estimated discrimination abilities for all/treatment-specific AEs are: 3.6/5.3; 4.6/5.7; 5.0/6.1; 5.6/4.8; 5.4/6.9. Treatment separation is better for treatment-specific AEs except the quantile regression. For AEBS, the quantile regression is the best for all AEs. For AE rate, the negative binomial model is good for treatment-specific AEs.

For both trials the Tweedie model has the best fit (AIC).

For future CTs, we recommend AEBS as a versatile and sensitive metric to measure AE burden. Based on the results from the two trials above, Tweedie regression yields good treatment discrimination ability and model fit due to its statistical flexibility to handle frequent zero values and the explicit correlation between mean and variance. A simulation study will be conducted to evaluate whether Tweedie regression is superior to other models under different scenarios.

After the release of a drug on the market, pharmacovigilance monitors the occurrence and changes in known adverse drug reactions (ADRs) as well as detects new ADRs in the population. This is done to keep a drug‘s harm profile updated and can potentially result in adjustments of the prescription labeling or – in the extreme case – a recall of the product from the market. In recent years the interest in the use of longitudinal electronic health records for pharmacovigilance increased. Cornelius et. al (2012) provided a signal detection test based on the Weibull distribution shape parameter. Sauzet and Cornelius (2022) refined this approach, proposing a test based on the power generalized Weibull distribution shape parameters (PgWSP). The power generalized Weibull (PgW) distribution is characterized by a scale parameter and two shape parameters. If both shape parameters of the PgW distribution are equal to one, the distribution reduces to an exponential distribution with constant hazard over time. A constant hazard is interpreted as no temporal association between a drug and an adverse event.

Signal detection can be improved by incorporating existing knowledge about the ADR profile of drugs from the same family or based on expert knowledge about the drug mechanism. Therefore, we propose the development of a Bayesian PgWSP test. The hypothesis test in the Bayesian context is based on a region of practical equivalence (ROPE) reflecting the shape parameters’ values under the null hypothesis (Krushke, 2015). For each parameter, a credibility interval deducted from the posterior density is compared to the ROPE. If the intersection between ROPE and credibility interval is empty for at least one shape parameter, a signal is raised.

We performed a simulation study with the aim to find the optimal ROPE and credibility interval for signal detection using a Bayesian PgWSP approach. Samples are generated with varying sample sizes, background rates reflecting the number of symptom observations in the general population, and proportions of adverse event observations caused by a drug. For the scale parameter, we test a fixed prior value, a gamma, and a lognormal prior distribution. For both shape parameters, either gamma or lognormal prior distributions are used. The priors are characterized in terms of mean and standard deviation based on hypothetical prior assumptions. Prior assumptions considered are: the symptom is no ADR of the drug, the symptom is an ADR of the drug with the highest risk of occurrence either at the beginning, in the middle, or at the end of the observation period. ROPEs of various types or widths are considered. For posterior credibility intervals, we use either equal-tailed or highest-density intervals of the posterior distributions with varying credibility levels. The optimal tuning parameters are determined based on the area under the curve.

This is the first part of a tandem presentation on the topic of the evaluation of adverse events by the Cox proportional hazards regression in early benefit assessment. The second part will be presented at the GMDS Annual Conference in Heilbronn, Germany, 2023.

For the early benefit assessment of drugs in Germany, the pharmaceutical company must describe the extent of an added benefit of the drug to be assessed compared with an appropriate comparator therapy [1]. The confidence interval of a significant effect must lie completely outside a certain corridor around the null effect for the extent of the effect to be regarded as minor, considerable or major. The corridors are defined by different thresholds depending on outcome category (e.g. all-cause mortality, quality of life or adverse events). For endpoints in the category of adverse events, frequently no events in one of the arms are observed, and while the log rank test provides appropriate p-values, the standard Cox proportional hazard regression does not provide valid effect estimates with corresponding confidence intervals. Thus, in the case of a statistically significant effect according to the p-value, the extent of the effect cannot be determined. Consequently, the overall assessment of the early benefit might be hampered.

Heinze and Schemper proposed an adaption of the Firth correction to reduce bias from maximum likelihood estimation for the Cox proportional hazard [2].

To assess the applicability of this approach, we performed a simulation study of time to event analyses with zero events. We will present results from this study and discuss the situations, in which the application of the Firth correction provides reliable estimates that can be used for the assessment of the extent of an added benefit.

In the second part of the tandem presentation, we will discuss the current procedure for time-to-event analysis with zero events and the possible contribution of the Firth correction in the early benefit assessment of adverse events by the IQWiG.

1. IQWiG. General Methods 6.1 [online]. 2022. URL: https://www.iqwig.de/en/about-us/methods/methods-paper/
2. Heinze and Schemper (2001). Biometrics 57(1):114–119.