Missing Data and Quality

The one idea: A missing value is not a zero and not a normal value — it is the absence of one, and your tools quietly skip, drop, or misfill it in ways that silently change your numbers. Before trusting any average or count, find out how much is missing, why it's missing, and what your tools did with it.

⚠️ The trap: "A few blanks in the column — I'll just fill them with 0 (or let the average skip them) and move on." That feels right and it's wrong. Missing is not zero, and a skipped NULL quietly shrinks your denominator to a biased subset — find out which of the three stories applies (never collected, lost, not applicable) and report the count behind every average.

Every real dataset has holes. Customers without birthdays, orders without regions, sessions without referrers. The question is never whether your data has missing values — it's whether you know where they are, why they're there, and what your tools silently do with them.

And that is the dangerous part. Missing values don't sit there inertly; they actively change your numbers. Aggregations skip them without telling you. Joins drop them. Comparisons involving them return neither true nor false. An analyst who doesn't profile missingness first isn't computing on the dataset they think they are — they're computing on the subset their tools quietly selected for them.

NULL and NaN: absence is not a value

In SQL, a missing value is NULL; in pandas, it's NaN (or NaT for dates, None for objects). Different names, same core semantics: it's not a value, it's the absence of one, and it behaves unlike…