Nlp Fundamentals and Bag of Words

NLP Fundamentals and the Bag-of-Words Model

The one idea: Turn each piece of text into a vector of word counts, throw away everything about grammar and word order, and feed the vectors to an ordinary classifier. This "bag-of-words" approach is dumb by design — and it gets roughly 85% of the way to a working text classifier anyway. Your job is knowing what that last 15% is made of.

⚠️ The trap: "not good" and "good" produce nearly identical bag-of-words vectors — both contain the word "good," and a plain word-count model has no concept of "not" negating it. If your ticket/review classifier is quietly flipping negated sentences, this is why, and no amount of more training data fixes it — you need a different feature (bigrams, or a model that reads in order) to close that gap.

Every text classification system — spam filters, support-ticket routers, review sentiment scores — starts from the same problem: text is a string, and models need numbers. The path from one to the other is what this lesson covers: tokenization (splitting text into pieces), bag-of-words (turning those pieces into a numeric vector), TF-IDF (a smarter weighting of that vector), and a real scikit-learn pipeline that strings it all together into a working classifier.

Tokenization is harder than .split(' ')

The obvious first move is to split text on spaces:

text = "I can't believe the shipment didn't arrive, again!!"
text.split(' ')

["I", "can't", "believe", "the", "shipment", "didn't", "arrive,", "again!!"]

Two problems are visible immediately. "arrive," and "again!!" still have punctuation glued to them — a model will treat "arrive," and "arrive" (from some other sentence)…