Evaluating and Selecting Regression Models

The one idea: No single regression metric tells the whole story — R² tells you how much variance you explained, RMSE and MAE tell you how big your typical error is in real units, and none of them, computed on the data you trained on, tell you anything about how the model will do on data it hasn't seen. Model selection has to be judged on held-out performance, not training-set fit.

⚠️ The trap: Training-set R² can only go up (or stay flat) as you add predictors — including predictors that are pure random noise. A model with a higher training R² than a simpler alternative is not necessarily the better model; it may just have more knobs to memorize training noise with. Compare models on cross-validated or held-out test performance, never on training-set R² alone.

The metrics, and what each one actually answers

Every regression metric below is computed from the same raw ingredient — the residuals, y_actual - y_predicted — but they answer different questions.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error

np.random.seed(7)
n = 80
sqft = np.random.normal(2000, 500, n)
bedrooms = np.random.randint(2, 6, n)
age = np.random.randint(0, 60, n)
price = 50_000 + 120 * sqft + 8_000 * bedrooms - 900 * age + np.random.normal(0, 25_000, n)

X = pd.DataFrame({'sqft': sqft, 'bedrooms': bedrooms, 'age': age})
y = price

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
model = LinearRegression().fit(X_train, y_train)
pred_test = model.predict(X_test)

r2 = r2_score(y_test, pred_test)
rmse = mean_squared_error(y_test, pred_test) ** 0.5
mae = mean_absolute_error(y_test, pred_test)

print(f"R^2:  {r2:.3f}")