Estimating the Performance of Predictive Models with Resampling Methods

Machine learning methodology is gaining popularity in psychology and other social sciences. One core element is how to estimate the accuracy of individual predictions when applying a predictive model to new observations in practice. In this talk, Dr. Florian Pargent will give an introduction on how such an evaluation of predictive performance is achieved by resampling methods like k-fold cross-validation. Specifically, he will demonstrate why such out-of-sample estimates of predictive performance are necessary when working with machine learning algorithms (e.g. Random Forests). Then, it will be discussed whether resampling methods should be more frequently used when evaluating linear regression models which are predominantly used in psychological science. Finally, Dr. Florian Pargent will outline common pitfalls in evaluating model performance, that can lead to gross overestimates of predictive accuracy in the literature when preprocessing steps like variable selection are not adequately combined with the resampling scheme.