Support Vector Machines

The one idea: A Support Vector Machine doesn't just look for a line that separates two classes — it looks for the line with the widest possible cushion (margin) on both sides. Only the points closest to that cushion (the "support vectors") matter for where the line sits; a point sitting confidently in the middle of its own class's territory could vanish from the training set and the boundary wouldn't move an inch.

⚠️ The trap: Cranking up gamma on an RBF-kernel SVM makes training accuracy climb toward 100% — and it feels like the model is "working better." It isn't. High gamma means each training point only influences a tiny neighborhood around itself, so the boundary starts wrapping tightly around individual points, including noisy ones. You get a boundary that memorizes the training set and falls apart on new data — the textbook definition of overfitting, arriving disguised as a good training score.

Maximum margin: why "any separating line" isn't good enough

Imagine two classes of points that are cleanly separable — you could draw dozens of different straight lines between them and every one would get 100% training accuracy. Which one should you pick? A perceptron (an early linear classifier) stops as soon as it finds any line with zero training errors, which can leave that line hugging one class closely, with almost no breathing room before it starts misclassifying new points.

An SVM instead picks the line that maximizes the margin — the distance from the boundary to the nearest point of either class. Wider margin means more room for a new, slightly-different point to still land on the correct…