Capstone Audit an Analysis Notebook

The one idea: You can catch a notebook's worst errors by procedure, not luck — restart and re-run it, follow the row counts, hand-check one number, then read the conclusion against the corrected table. Here fifteen minutes of that reading kept a fifth of a budget from moving to the weakest channel, and every bug had run without a single error.

⚠️ The trap: The table is tidy, the numbers are specific, and the reasoning flows, so a notebook this polished must be safe to present. That feels right and it's wrong. Every one of the three bugs here ran without a single error and produced a plausible table — only the procedure (restart, follow the row counts, hand-check one number) reveals that the recommendation actually points backwards.

A colleague forwards a notebook titled "Q2 Campaign Performance" with the note: "AI helped me put this together — conclusion is at the bottom, planning to present Thursday." The recommendation moves real budget. Your job: decide whether to trust it, using nothing but the reading skills from this track. There are three bugs in here. Let's find them the way an auditor would — by procedure, not luck.

The notebook as received

In [8]:  import pandas as pd
         spend = pd.read_csv("q2_spend.csv")          # date, channel, spend
         conv  = pd.read_csv("q2_conversions.csv")    # date, channel, visits, conversions
         print(len(spend), len(conv))
         273 364

In [2]:  conv['cvr'] = conv['conversions'] / conv['visits']

In [9]:  df = conv.merge(spend, on=['date', 'channel'], how='inner')
         len(df)
Out[9]:  273

In [4]:  summary = df.groupby('channel').agg(
             spend=('spend', 'sum'),
             conversions=('conversions', 'sum'),
             cvr=('cvr', 'mean'),
         )
         summary['cpa'] = summary['spend'] / summary['conversions']
         summary
Out[4]:
                  spend  conversions    cvr   cpa
         channel
Email 18200 520…