Kernel PCA

The one idea: Regular PCA can only draw straight lines (or flat planes) through your data — it looks for the directions of maximum variance and nothing more. When the real structure in your data is curved or looped, no straight line separates it, no matter how you rotate the axes. Kernel PCA fixes this by first measuring how similar every pair of points is through a nonlinear "kernel" function, then running PCA on those similarities instead of on the raw coordinates — effectively letting it draw curved boundaries without ever explicitly computing the curved space.

⚠️ The trap: Reaching for kernel PCA by default because "nonlinear sounds more powerful" is expensive and often unnecessary. Kernel PCA doesn't give you a reusable projection matrix like regular PCA does — it has to keep the entire training set around to project any new point, and it has one more hyperparameter (gamma, or the kernel choice) that you have to tune correctly or the result is garbage. If a scatter plot or a quick PCA-then-classify baseline already looks linearly separable, kernel PCA is very likely solving a problem you don't have.

Why regular PCA fails on curved data

PCA's whole mechanism — eigenvectors of the covariance matrix — only ever produces straight-line directions. It finds the axis of maximum spread and projects onto it. That works great when your classes or clusters differ along some straight direction. It fails completely when the structure that separates your data is a shape, not a direction.

Worked example: concentric circles

Generate a classic toy dataset — an inner circle of points, surrounded by an outer ring, with two…