We enhance the multi-armed bandit model by considering endogenously non-stationary rewards – specifically rewards that are parametric functions of policy histories (learning). We further incorporate queueing costs, fairness, and arm correlation. We propose the L-UCB, FL-UCB, and QFL-UCB algorithms to solve our model, prove its logarithmic regret, and apply it to split-liver transplantation.

We adapt online learning techniques to scenarios with time-to-event data, where there is a delay between choosing an arm and observing feedback that is endogenous to the quality of the arm. We posit a multi-armed bandit algorithm with a cox-proportional hazards estimator, prove guarantees on the regret under this algorithm, and analyze its performance on a dataset of metastatic breast cancer clinical trials, comparing it to that of other adaptive allocation schemes.

Lack of treatment adherence significant barrier to reducing the global disease burden of tuberculosis (TB). We study the design of targeted interventions for a treatment adherence support platform running in Kenya, whose goal is to help patients on TB treatment. We show empirically that there is large heterogeneity in treatment effects of interventions, and we devise a novel online learning policy based on Thompson Sampling that significantly outperforms the currently employed policy.