K-fold cross-validation

If we don’t have enough training examples, we could use K-fold cross-validation (CV):

• Split data into $k$ equal subsets / folds to test on
• Train on $k-1$ sets and test on remainder fold only
• Repeat $k$ times (test on each fold only once)
• Computer misclassification error averaged over all $k$ test folds

The final performance is the average of the performances for each fold. Average test set score is a better estimate of the error rate than a single score. Common values of $k$ are $5$ and $10$, both giving error estimates that are very likely to be accurate.

Extreme case is when $k=n$, the number of data points.