Word Embeddings and Semantic Similarity

The one idea: Bag-of-words represents each word as a column in a giant sparse table, with no notion that "good" and "great" have anything to do with each other — they're just two unrelated dimensions. Word embeddings instead represent each word as a short, dense vector of a few hundred numbers, trained so that words used in similar contexts end up geometrically close together. The payoff: a model can now generalize from "the food was great" to "the food was fantastic" even if it only ever saw one of those two sentences during training.

⚠️ The trap: The famous king - man + woman ≈ queen demo makes embeddings look like they understand meaning. They don't — they've only learned that words appearing in similar surrounding contexts get pushed to similar places in the vector space. That's a powerful, useful regularity, but it's a statistical side-effect of co-occurrence patterns, not comprehension, and it fails unpredictably on relationships the training corpus didn't happen to encode well (rare words, sarcasm, negation, anything culturally specific to a corpus the embeddings weren't trained on).

Starting from where bag-of-words stops

The bag-of-words lesson built a vocabulary and represented each document as a vector of word counts over that vocabulary — one column per distinct word, mostly zeros, no relationship between columns. That representation has a specific blind spot worth stating precisely: in a bag-of-words vocabulary, "good" and "great" are two entirely unrelated columns. A classifier trained on documents containing "good" gets zero transfer to a new document that happens to say "great" instead, even though a human reader would treat them as near-synonyms in most contexts.…