Understanding the concentration-response relationship of a candidate drug is one of the main goals in toxicology and especially drug development. Therefore, several authors emphasized the importance of optimal designs regarding concentration-response experiments. Using classical optimal design approaches significantly enhances the precision of the estimated concentration-response curve or rather specific parameters. Thus, optimal designs can substantially improve comprehension in toxicological context.
We extend classical approaches of optimal design of experiments (see [Pukelsheim] among many others) such that they can be applied for the analysis of concentration-response relationships in the context of gene expression data. The experimental conditions of such data are new challenges in planning those experiments, since all genes are elevated simultaneously. Thousands of concentration-response relationships need to be determined, so the key question is which design
works best for the simultaneous analysis. Thereby gene expression data of valproic acid applied to human embryonic stem cells is analyzed to compare different designs [Krug et al.]. First of all genes with biologic activity are evaluated with the Multiple Comparison Procedure and Modelling approach [Bretz et al.]. Simultaneously the true underlying concentration-response relationships are fitted using separate sigmoid Emax models for all active genes. Then locally D-optimal designs are identified for every considered gene. Based on the locally D-optimal design, a summarized design is established using the K-means algorithm. Moreover, a D-optimal design for simultaneous inference is developed by using an appropriate Bayesian D-optimality criterion. Both developed designs are compared to the design, originally used in the experimental set-up, an equidistant and log-equidistant design with respect to their D-efficiencies. Moreover, a simulation study is conducted to demonstrate the differences of all designs practically. In that simulation study, we also vary the total sample size to investigate its influence on the precision of the model fits.

The results actively demonstrate to support the consideration of using optimal design approaches for gene expression data. Especially the D-optimal design for simultaneous inference and the design developed with K-means perform considerably better than the originally used design and the log-equidistant design, both in terms of the theoretical and the practical comparison. Measured by the root mean squared error (RMSE) the original, equidistant and log-equidistant design have an
inferior performance in the simulation study compared to the two other designs. Besides the precision of the model fits highly increases by enlarging the total sample size. Thus, it is recommendable using as many observations as possible, whereupon minimal 27 data points should be used in this analysis. Summarizing, the D-optimal design for simultaneous inference leads to the most exact model fits, consequently, it should be preferred to the other designs investigated.

[Bretz et al.] Bretz, F., Pinheiro, J. C. and Branson, M. (2005): Combining multiple comparisons and modeling techniques in dose-response studies, Biometrics, 61(3), p. 738-748.

[Pukelsheim] Pukelsheim, F. (2006): Optimal Design of Experiments, Wiley, New York.

[Krug et al.] Krug, A. K. et al. (2013): Human embryonic stem cell-derived test systems for developmental neurotoxicity: a transcriptomics approach. Arch Toxicol, 87, p. 123–143.

A proper experimental design is key for the validity of statistical results and their interpretations. In pharmaceutical industry research and preclinical experiments build the starting point for further developments and corresponding results are usually based on small sample sizes. One important goal is to determine a precise estimation of the dose-response shape of the test item. This includes a minimal variance of the calculated model parameters leading to a minimal sample size to obtain a desired precision which also takes the “Three Rs Principle” (Replacement, Reduction, Refinement) into consideration.
In this talk, d-optimal design settings to improve the efficiency of an experimental design are discussed and various non-clinical applications are presented. A particular focus is on the practicability as well as the robustness of the statistical outcomes based on moderate model misspecifications. Therefore, optimal design settings are compared to balanced designs in terms of the precision of the estimates, and bayes optimal designs are investigated to account for the underlying model uncertainty during the planning stage. In addition, a multivariate approach, e. g. for the analysis of drug combinations, will be investigated.

Applying 10 simple rules for good research practice in pre-clinical research

The reproducibility or lack thereof has been identified as a focus area for science with one of the headlines “More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments [1]. There has been much discussion of potential solutions including Recommendations to improve study design and analysis in an NIH White Paper [2]. In this presentation the ideas listed in recently by Schwab et al. [3] will be reviewed and their application within pre-clinical research assessed.

1. Baker, M (2016). Is there a reproducibility crisis? Nature, 533; 452-454. https://www.nature.com/news/polopoly_fs/1.19970!/menu/main/topColumns/topLeftColumn/pdf/533452a.pdf
2. Report of the ACD WORKING GROUP ON ENHANCING RIGOR, TRANSPARENCY, AND TRANSLATABILITY IN ANIMAL RESEARCH, June 11^th 2021, https://acd.od.nih.gov/documents/presentations/06112021_RR-AR%20Report.pdf
3. Schwab S, Janiaud P, Dayan M, Amrhein V, Panczak R, Palagi PM, et al. (2022) Ten simple rules for good research practice. PLoS Comput Biol 18(6): e1010139. https://doi.org/10.1371/journal.pcbi.1010139

Background:

The planning and analysis of preclinical animal trials is challenging because of small sample sizes and sparse prior data. Standard analysis results (e.g. from frequentist t-tests) are possibly of limited validity in calculating an appropriate sample size for the actual animal experiment. So alternative approaches are desired such as (i) a rearrangement of prior data to a final full factorial design or (ii) the incorporation of prior knowledge in Bayesian analysis to get a meaningful number of necessary animals.

Methods:

For the first approach, based on historical data of two groups, possible full factorial designs were created. Therefore different interaction patterns were presumed and sample sizes for the potential main and interaction effects were calculated. For the second approach, different concepts for design analysis with Bayesian methods were compared and evaluated with respect to their applicability in preclinical, translational research.

Results:

If a full factorial design is planned, the interaction effects should always be taken into account. Besides a gain in information, also a smaller sample size is possible. Bayesian methods allow a better representation of uncertainties in the model parameters. Using design analysis as basis for a sample size decision, instead of the classical sample size calculation by solving power equations may be preferred since it reflects the alignment to additional design goals and the sensitivity of the estimation results to the model assumptions.

Discussion:

As with a small group size, the usage of parametric test assumptions may not be valid. Moreover, current limitations reside in the complexity of Bayesian methodology, which make it challenging to understand and control the impact of single design components on estimation results, and in the limited availability of historical animal experiment data.

Pre-clinical studies are characterized by the measurement of multiple outcomes in small samples. To analyze these outcomes often rank-based or resampling permutation or bootstrap non-parametric test are applied. Recently, the non-parametric Generalized Pairwise Comparisons (GPC) methodology has attracted a lot of attention in clinical trials for its ability to evaluate multiple outcomes. Moreover, the GPC permutation test enjoys good small-sample properties. Curiously, this permutation test does not require re-sampling, a feature that can be extended to GPC bootstrap-based inference. GPC extends the Mann-Whitney test to multiple outcomes and shows great flexibility for the design of a study. It allows for any number and type of outcomes, allows for prioritizing the outcomes by clinical severity, allows for matched designs, allows for adding a threshold of clinical relevance and accounts for the correlation between the outcomes. In this talk, the generalized pairwise comparison ideas and concepts for small sample trials are introduced and critically evaluated for their applicability in preclinical trials.