Evaluating and Selecting Classification Models

The one idea: Accuracy answers "what fraction of predictions were correct" — and on any dataset where one class is rare, that question has a boring, misleading answer: predict the majority class every time and rack up a high score without learning anything. The confusion matrix, precision, recall, and ROC/AUC exist because "correct how often" is the wrong question the moment your classes are imbalanced or your two error types cost different amounts.

⚠️ The trap — the accuracy paradox. A fraud model that never once predicts fraud can post 99%+ accuracy if fraud is rare enough. The number looks like a win in a slide deck and is actually a model that does nothing. This is the single most common way classification results get misread by non-technical stakeholders, and it's the centerpiece of this lesson — worked in full detail below.

The confusion matrix: the one table everything else is built from

Every classification metric in this lesson is computed from four counts, laid out in a 2x2 table comparing predicted label to actual label:

from sklearn.metrics import confusion_matrix
import numpy as np

# 10,000 transactions, 100 are actually fraud (1%)
# Model predicts fraud on 40 of them; 30 of those 40 are genuinely fraud
y_true = np.array([1]*100 + [0]*9900)
y_pred = np.array([1]*30 + [0]*70 + [1]*10 + [0]*9890)  # 30 TP, 70 FN, 10 FP, 9890 TN

cm = confusion_matrix(y_true, y_pred)
print(cm)