Cross Validation and Grid Search

The one idea: A single train/test split gives you one noisy sample of "how good is this model" — reshuffle the split and the number moves. K-fold cross-validation averages over many splits to get a stable estimate, and it's also the honest engine for tuning hyperparameters — as long as the final test set stays locked away until the very end.

⚠️ The trap: We kept adjusting the model until test accuracy hit 94% — that feels like rigor and it's actually the crime. Every time a hyperparameter gets nudged in response to test-set performance, the test set stops being a test of generalization and starts being another training signal; the "final" number it eventually reports is inflated, sometimes badly.

You've built a classifier. You run train_test_split, fit it, score it: 91% accuracy. Respectable. Now split the same data again with a different random seed, fit the same model the same way: 87%. Split it a third time: 93%. Nothing about the model changed — only which rows happened to land in the test set. So which number is "the" accuracy? Trick question: none of them is trustworthy alone, and that instability is the whole reason cross-validation exists.

A single split is one roll of the dice

Here's the mechanism laid bare. train_test_split(X, y, test_size=0.3) draws a random 30% of your rows into a test set. If that 30% happens to be slightly easier than average — more of the "obvious" cases — your score is flattered. If it happens to include a disproportionate share of the hard, ambiguous cases, your score is punished. Neither outcome tells you about the model; both…