When fitting and selecting models in a data science project, how do you know that your final model is good? And how sure are you that it's better than the models that you rejected? In this Part 3 of our four part mini-series "How do you know if your model is going to work?" we develop out of sample procedures. Previously we worked on:
Out of sample procedures
Let's try working "out of sample" or with data not seen during training or construction of our model. The attraction of these procedures is they represent a principled attempt at simulating the arrival of new data in the future.
Hold out tests are a staple for data scientists. You reserve a fraction of your data (say 10%) for evaluation and don't use that data in any way during model construction and calibration. There is the issue that the test data is often used to choose between models, but that should not cause a problem of too much data leakage in practice. However, there are procedures to systematically abuse easy access to test performance in contests such as Kaggle (see Blum, Hardt, "The Ladder: A Reliable Leaderboard for Machine Learning Competitions").
The results of a test/train split produce graphs like the following: The training panels are the same as we have seen before. We have now added the upper test panels. These are where the models are evaluated on data not used during construction. Notice on the test graphs random forest is the worst (for this data set, with this set of columns, and this set of random forest parameters) of the non-trivial machine learning algorithms on the test data. Since the test data is the best simulation of future data we have seen so far, we should not select random forest as our one true model in this case - but instead consider GAM logistic regression. We have definitely learned something about how these models will perform on future data, but why should we settle for a mere point estimate. Let's get some estimates of the likely distribution of future model behavior.
To read more: Win-Vector blog - How do you know if your model is going to work? Part 3: Out of sample procedures