The statistical evaluation of developed models and algorithms is an essential part of applied machine learning and predictive modelling. This half-day course is suitable as a concise introduction or refresher for this important topic. It is divided into three parts with sufficient time for participant questions and breaks in between.

Initially, we will repeat essential machine learning basics and cover core concepts of model evaluation. We will mainly consider classification tasks and the most relevant assessment criteria (discrimination, calibration) but also summarize adaptations for regression and survival problems. In the main part, we discuss common pitfalls (leakage, multiplicity, ….) in model evaluation and appropriate best practices to avoid and/or rectify them. Finally, we touch upon some advanced topics and cover important practical aspects (software, reporting, reproducibility) that are required for a successful evaluation study.

The course contents are illustrated by means of real-world data examples, including R code to showcase how the numerical results were obtained. There are no explicit coding sessions in this short course, so a laptop is not necessarily required. The course materials will be made available so that participants have the opportunity to individually reproduce the numerical examples after the course.

Prerequisites:

• Initial practical experience in applied machine learning
• Basic knowledge of descriptive and inferential statistics