Naive Bayes and Tree Based Classifiers

The one idea: Naive Bayes makes a wildly unrealistic assumption — that every feature is independent of every other feature, given the class — and then works surprisingly well anyway, because for most classification tasks you only need to get the ranking of probabilities right, not the exact number. Decision trees make no such assumption; they just ask a sequence of yes/no questions about your features, one at a time, until the answer is clear.

⚠️ The trap: A single decision tree grown without a depth limit will hit 100% training accuracy — it keeps splitting until every leaf contains exactly one class. That's not a sign the tree found real structure; it's a sign the tree memorized the training set, including its noise. A tree that perfectly explains history is not the same as a tree that predicts the future. Always check unlimited-depth training accuracy against test accuracy before trusting it.

Naive Bayes: probability from counting, not fitting

Naive Bayes flips the direction most classifiers reason in. Instead of directly modeling "given these features, what's the probability of class 1," it uses Bayes' theorem to reason backward from "given class 1, how likely are these features" — a quantity that's often much easier to estimate by simple counting.

The setup. For a new email, you want P(spam | words in email). Bayes' theorem says:

P(spam | words) ∝ P(words | spam) × P(spam)

P(spam) is trivial — just the fraction of spam in your training data. P(words | spam) is the hard part... unless you make the "naive" assumption that every word's presence is independent of every other…