Regularization Ridge Lasso Elasticnet

Regularization: Ridge, Lasso, and ElasticNet

The one idea: Plain linear regression finds the coefficients that fit the training data best, full stop — and with enough features (especially correlated or irrelevant ones), "fits best" starts to mean "chases noise." Regularization adds a penalty for large coefficients to the cost function, trading a small amount of bias for a often-larger reduction in variance. Ridge, Lasso, and ElasticNet are three different shapes of that penalty, and the shape changes what the fitted model looks like — not just how well it generalizes.

⚠️ The trap: Ridge and Lasso are not interchangeable knobs for "how much regularization." Lasso can zero a coefficient out completely — dropping a feature from the model entirely — while Ridge only ever shrinks coefficients toward zero, never all the way there. Assume Lasso when you actually needed Ridge (or vice versa) and you'll either lose real predictive signal that Lasso zeroed out, or keep a bloated feature list that Lasso would have cleaned up for you.

Why plain regression overfits with many features

Ordinary least squares minimizes squared error and nothing else: J(w) = Σ(y - ŷ)². Nothing in that objective penalizes a coefficient for being large, so with enough features — especially more features than you have a comfortable amount of data for, or several features that are correlated with each other — OLS is free to assign big, offsetting weights to noise. Two nearly-identical columns can each get a huge coefficient of opposite sign that cancels out on the training data, and generalizes terribly, because there's nothing in the cost function telling the optimizer that's a bad idea.

Regularization fixes this by…