We consider different models for multivariate data, trying to impose as few assumptions as possible, in particular also allowing for endpoints that are not all measured on the same scale. Based on these models, different analysis methods are justifiable for inference. Some of them can be considered part of the class of generalized pairwise comparison (GPC) procedures. We discuss advantages and disadvantages of different approaches in terms of statistical performance, robustness, flexibility, and interpretability. Application of the methodology is illustrated with real data examples.

The method of Generalized Pairwise Comparisons (GPC) extends the Wilcoxon Mann-Whitney non-parametric statistical test from a single outcome to multiple outcomes hierarchically ordered for the comparison of two treatment groups (e.g., in a randomized clinical trial). The method estimates a benefit-risk metric called the “Net Treatment Benefit” (NTB), defined as the net probability of a better outcome in one treatment group than in the other. However, properly conveying GPC results is challenging because the method has been recently proposed, making its results unfamiliar and its interpretation not straightforward for clinical-trial stakeholders, including statisticians, physicians, and patients. Additional to its novelty, the multivariate nature of a GPC analysis, while considered a major strength both from a statistical and a clinical point of view, is also a source of challenges for communication around results. This presentation will share several novel ways of communicating GPC results and the NTB to different audiences, from intuitive visual aids that can be quickly understood by non-statisticians to more rigorous and exhaustive tables and figures.

Small samples occur in a variety of different areas and especially in pre-clinical research and translational trials. Most statistical procedures rely on asymptotic results and are thus applicable in large samples only. In case of small samples, they tend to not control the type-I error rate and over-reject the null hypothesis. In addition, postulating a certain data distributional model (e.g. normally distributed data) is often misleading. Nonparametric rank-based methods on the contrary do not rely on any distributional assumption and are thus applicable in these scenarios. However, which effects underlie such methods and which hypotheses are actually tested? In this talk, we discuss such methods in detail, focus on the use of Bootstrap, and (studentized) permutation tests as approximate solutions for small samples. Extensive simulation illustrate the applicability of the methods in (very) small samples. Real data sets illustrate the applications.

In many clinical situations, the medical question of interest requires the conduct of a non-inferiority trial (NI), but the latter are often unfeasible in addition to bringing several challenges in contrast to superiority trials. In this presentation, we examine the use of generalized pairwise comparisons (GPC) as a pragmatic alternative to NI trials for addressing the problem of ensuring efficacy and tolerability. The method of GPC is a recent proposal that allows the simultaneous evaluation of several outcomes of interest that can be of any type. These outcomes can further be prioritized to reflect one’s opinion regarding their perceived hierarchy of clinical importance. As an illustration, we consider the design of a randomized trial for patients with acute promyelocytic leukemia, where reducing the standard treatment dose is intended to improve tolerability. We highlight the different steps and choices for designing a trial constructed on GPC, based here in particular on historical data. The latter form the basis of a thorough simulation exercise for sample size determination. This presentation thus highlights how GPC can be considered an essential tool for assessing efficacy and tolerability in scenarios where NI trials are difficult to conduct, particularly when researching vulnerable groups.

Background: The Net Benefit (Δ) is a measure of the benefit-risk balance in clinical trials, based on Generalized Pairwise Comparisons (GPC) using several prioritized outcomes and thresholds of clinical relevance. We extended Δ to N-of-1 trials, with a focus on patient-level and population-level Δ.

Methods: We developed a Δ estimator at the individual level as an extension of the stratum-specific Δ, and at the population-level as an extension of the stratified Δ. We performed a simulation study mimicking PROFIL (NCT02050360), a series of 38 N-of-1 trials testing low and high-dose sildenafil versus placebo in Raynaud’s phenomenon on three outcomes, to assess the power for such an analysis with realistic data. We then reanalyzed PROFIL using individual-level GPC with expert-defined outcome hierarchy and validated minimal clinically important differences (MCID) acting as thresholds of clinical relevance. This reanalysis was interpreted in the context of the main analysis of PROFIL which was performed in a Bayesian framework and reported : i) individual probabilities of efficacy, and ii) individual adjusted risk variations. We showed a straightforward way to aggregate individual-level Δ in order to estimate a population-level Δ, while highlighting similarities and differences between our aggregation method and a more usual random-effects meta-analysis.

Results: Simulations under the null showed good size of the test for both individual and population levels. The test lacked power when being simulated from the true PROFIL data, even when increasing the number of repetitions up to 140 days per patient. PROFIL individual-level estimated Δ were well correlated with the probabilities of efficacy from the Bayesian analysis while showing similarly wide confidence intervals. Likewise, individual-level Δ led to similar conclusions than individual adjusted risk variations. Population-level estimated Δ was not significantly different from zero, consistently with the previous Bayesian analysis.

Conclusion: GPC can be used to estimate individual Δ which can then be aggregated in a meta-analytic way in N-of-1 trials. We argue that GPC ability to easily incorporate patient preferences (thresholds of clinical relevance on the same scale as the outcome itself and outcome prioritization) allow for more personalized treatment evaluation, while needing much less computing time than Bayesian modeling. Finally, we discuss the current limits of GPC usage in N-of-1 trials and some ways to alleviate them, as well as undergoing developments.