Handling Missing Data

The one idea: Every row you drop because of a missing value is a row you're betting was missing at random. That bet is usually wrong — data goes missing for reasons (a sensor that fails under load, a customer who skips the income question because it's high or low), and dropping those rows quietly reshapes your dataset toward whoever answered everything.

⚠️ The trap: df.dropna() runs without complaint and returns a smaller, "clean" DataFrame — nothing about the output tells you why those rows were missing, or that the rows you kept are no longer representative of the rows you started with. A model trained on the survivors learns the survivors' pattern, not the population's.

Real datasets have holes in them: a form field left blank, a sensor reading that failed to log, a join that didn't find a match. Before you can train a model, you have to decide what to do with those holes. There are really only two moves — drop the row/column, or impute (fill in) a value — and picking wrong is one of the most common ways a beginner quietly biases a model before any algorithm even runs.

Seeing the problem

import pandas as pd
from io import StringIO

csv_data = '''age,income,years_employed,approved
25,42000,2.0,1
41,,9.0,1
33,58000,,1
29,,1.0,0
52,91000,25.0,1
38,,4.0,0
'''
df = pd.read_csv(StringIO(csv_data))
print(df)

   age   income  years_employed  approved
0   25  42000.0             2.0         1
1   41      NaN             9.0         1
2   33  58000.0             NaN         1
3   29      NaN             1.0         0
4   52  91000.0            25.0         1
5   38      NaN             4.0         0

df.isnull().sum() tells you where the holes are:

print(df.isnull().sum())

age               0
income            3
years_employed    1
approved          0