Simple and Multiple Linear Regression

The one idea: Linear regression finds the one straight line (or flat plane, in higher dimensions) that minimizes the total squared distance between your data points and the model's predictions. Every coefficient it hands back is a "holding everything else constant" number — and the moment two of your predictors move together, that number stops meaning what it looks like it means.

⚠️ The trap: Adding more predictors to a linear regression always makes R² go up (or stay flat), even if the new predictor is random noise. A rising R² after adding a variable is not evidence that variable matters — it's a mathematical guarantee. Judge added predictors by their effect on test-set error, not by whether training R² improved.

Linear regression is the oldest model in the ML toolbox, and it's still one of the most used — not because it's the most powerful, but because it's fast, interpretable, and a strong baseline you should always fit before reaching for anything fancier. If a random forest can't beat a well-built linear regression, that's a real finding, not a footnote.

The mechanics: fitting a line nobody drew

Simple linear regression models one target variable y as a function of one input x:

y = w0 + w1*x

w0 is the intercept (where the line crosses the y-axis), w1 is the slope (how much y changes per unit of x). "Fitting" the model means picking w0 and w1. Ordinary Least Squares (OLS) — the default way scikit-learn fits LinearRegression — picks the w0, w1 that minimize the sum of squared residuals: the vertical distance between each actual point and…