The feature makes money.
The churn it looked to cost is not there.
First question: can the result be trusted at all?
A lift is only worth reading once assignment, coverage and data quality check out. Four gates, all cleared before any metric was compared.
Revenue moved, and it kept moving
Daily revenue per active user runs noisy, so the honest read is the accumulated gap. It widens steadily instead of arriving in one spike, which rules out a single promotion day.
Revenue per daily active user
ARPDAU by calendar day, both groups
Accumulated revenue gap
Group B total minus group A total, running
How sure are we about the size of the lift?
Revenue is heavily right skewed, so a t interval alone would be optimistic. Resampling both groups puts a range on the difference rather than a single number.
Bootstrap distribution of the ARPU difference
95% confidence interval for the per-user revenue difference. The whole interval sits above zero, so the direction is not in question.
of resamples put group B ahead. The uncertainty is about how large the gain is, not whether it exists.
The gain came from existing payers, not from new ones
This is the part that changes what the feature actually is. Nobody new started paying. The people already paying paid more, at every point of the spending distribution.
Spend per payer by percentile
A shift across the whole curve, not a few large accounts
What looked like the cost
Early retention drops for a full week before the curves meet again. My first readout called that churn and held the release. It is not churn: the next section shows why.
Cohort retention by day
Share of each group still active on day N
How I know it is not churn
Whether players left needs no cohort anchor at all. The count of active days per player sits in the published user level file, and it answers the question directly.
Share of players by activity
Cumulative, out of 59 days
| Active days | A | B | Diff |
|---|---|---|---|
| 3 or fewer | 4.660% | 8.952% | +4.29 pp |
| 5 or fewer | 12.291% | 14.516% | +2.23 pp |
| 7 or fewer | 15.087% | 15.051% | -0.04 pp |
| 14 or fewer | 18.941% | 18.672% | -0.27 pp |
Where the difference sits
Band by band, difference in percentage points
| Active days | A | B | B - A |
|---|---|---|---|
| 1 | 0.435% | 0.950% | +0.52 |
| 2 | 1.259% | 3.183% | +1.93 |
| 3 | 2.966% | 4.818% | +1.85 |
| 4 | 4.058% | 3.892% | -0.17 |
| 5 | 3.573% | 1.673% | -1.90 |
| 6 | 1.968% | 0.449% | -1.52 |
| 7 | 0.828% | 0.086% | -0.74 |
The cumulative shares converge by day seven. The disengaging segment holds 7,528 players in A and 7,541 in B: the same group, to within four hundredths of a point. Inside it, mean active days are 4.21 against 3.26. B removes nobody. It makes the same players stop about a day sooner, and those players have three to four active days out of fifty-nine.
Day 14 differs by 0.08 points, day 28 by 0.00, no player in either arm has zero active days, and daily actives trend upward in both arms across the whole window. Reproduce it with python analysis/reaudit_active_days.py in the repository: standard library only, runs on the committed data.
The call
My first readout held the release. Revenue was up, early retention was down, and trading a durable asset for a measurable one looked like a bad deal. The re-audit removed the durable asset from the equation: the same share of players disengages either way, and B only changes when the lowest-value segment stops.
What remains is a nine percent revenue gain with identical long-run retention. Across the window the treatment arm earned about twenty-six thousand dollars more. There is no measured cost to set against it.
Use day 14 and day 28 retention as the stopping rules, not day 1. Day 1 moves here for reasons that do not predict survival.
Rebuild the cohort metric on install date rather than first active day in the window, and censor mid-window installs. The conclusion holds without it, but the curve should be right.
Check whether players who now stop at day three instead of day five carry residual value. On this dataset they do not, but that is a property of the data, not a law.