Finding the Story

Analyzing Cheatsheet

Distributions first

• Look at the shape before reporting any summary — histogram or p50/p90/p99 (percentiles: p50 = the median, p90 = the value 90% of points fall at or below), one query
• Business data (revenue, latency, duration, engagement) is long-tailed by default
• Mean far above median = long right tail; the mean describes no one
• Choose the summary by the question:

Outliers

• Investigate before deleting — an outlier is one of: an error (fix, disclose), a real atypical event (whale; include/exclude per the question), or the story (fraud, bugs, new segment)
• When a metric jumps, sort by contribution and check the top 10 rows before crediting broad behavior change
• Handling: report medians/percentiles (best first move) \| trim top/bottom k% (disclose) \| winsorize (cap at p99, keep rows)
• Iron rule: disclose the treatment and show results with AND without the extremes

Time series

• Every series = trend + seasonality + noise; attribute the movement before attributing a cause
• WoW (week-over-week): cancels weekly cycle, fast detection, blind to annual patterns
• YoY (year-over-year): cancels annual cycle, measures true growth, slow, breaks on abnormal base years
• 7-day moving average cancels the weekly cycle (one of each weekday); costs lag — distrust the smoothed line's last points
• Never read a trend from two points: need 8–12 periods of history, movement outside normal wiggle, ideally 3+ consecutive periods

Partial periods

• Never compare an incomplete period to complete ones
• Best: period-to-date vs prior period-to-date (June 1–18 vs May 1–18)
• Acceptable: run-rate projection, clearly labeled
• Know your pipeline lag — grey out the last 1–2 days of operational charts

Cohorts

• Cohort = group by START time, follow forward on a time-since-start clock
• Read across a row = lifecycle decay; down a column = product change across cohorts
• Never average or zero-fill the unfinished bottom-right triangle
• Blended retention confounds product quality with acquisition mix — when mix shifts, the blended number can move opposite to every cohort

Funnels

• Step conversion (of those who reached N) = diagnostic view; overall conversion = product of all steps = business view
• Improvements multiply: lifting the worst step lifts overall proportionally — fix the 20% step, not the 85% step
• Build on a fixed entry cohort followed forward, with stated step order and time window — not same-week ratios of different populations
• Survivorship: any metric computed only on completers describes survivors ("median time to buy: 9 min — among purchasers"); push the denominator back to the population in the claim

Designing metrics that don't lie

• Every metric is a proxy (a stand-in) for something you can't measure directly — trust it only if moving the proxy reliably moves the true outcome, even when someone tries to move the proxy on purpose
• Use three roles together: north-star (slow, hard-to-game value, e.g. net revenue retention) \| input (fast weekly lever, e.g. tickets closed — most gameable) \| guardrail (must-not-get-worse, e.g. reopen rate, satisfaction)
• Goodhart's law: when a measure becomes a target, it stops being a good measure — a fair description degrades once people are rewarded for moving it
• Never ship an input metric with no guardrail — ask "what's the laziest way to hit this number without doing the real work?" and the answer names the missing guardrail
• Leading (fast, gameable) vs lagging (honest, late): steer by leading metrics but keep a lagging metric + guardrail in view, so a fake win can't run unchecked for a quarter
• Optimizing one metric alone (engagement, closures, clicks) buys its gains by degrading something unmeasured (retention, trust) — name what must not break before setting the target

Metric-movement triage (in order)

1. Data: partial period? pipeline lag? duplicate load?
2. Composition: mix shift across segments/cohorts?
3. Concentration: a handful of rows (whale, outage) driving it?
4. Seasonality: did the same thing happen a year ago?
5. Only then: a real behavior change worth a cause