Measuring the impact of a change

You rewrote your welcome emails, and the next month fewer new members cancelled. You raised your price, and sign-ups dropped. Did the change cause the result? Maybe. But membership numbers move for all sorts of reasons, and the month after a change is also the month after everything else that happened. Owners who read every before-and-after as cause and effect often end up reversing good decisions and doubling down on bad ones.

Measuring the impact of a change properly does not require statistics training. It requires a little discipline before, during and after the change.

The trouble with before and after

The simplest comparison is the month before against the month after. It is also the most easily fooled. Several things can create a difference that has nothing to do with your change:

• Seasonal patterns. Many memberships have busy and quiet times of year. A change made just before a quiet period can look like a failure.
• Other changes at the same time. A new email sequence launched in the same month as a promotion or a guest appearance cannot be judged on its own.
• Different members. A promotion or a new traffic source can bring in a different kind of member, who behaves differently regardless of what else you changed.
• Bouncing back from an unusual month. If you made a change because a number looked terrible, it might well have improved anyway, since unusually bad months are often followed by more normal ones. This is called regression to the mean.
• The novelty effect. Something new, such as a redesigned member home page, can attract attention at first and then fade.
• Small numbers. On a small membership, a difference of a few members is often just chance.

Write it down before you change anything

The most valuable habit is also the simplest: before making a change, write a short note. It forces you to decide what success looks like before you see the result, which stops you finding success in whatever number happens to rise. Here is a note from a hypothetical paid trade newsletter for small-farm growers:

Change: Replace the three-email welcome sequence with a five-email sequence that walks new subscribers through the archive.

Why: Too many new subscribers cancel in their first two months.

Main measure: Share of new subscribers still paying 60 days after joining.

Baseline: Average of the last six monthly cohorts, with the highest and lowest.

Expected result: A clear rise in 60-day retention across the first three cohorts.

Judge on: When the third cohort after the change reaches 60 days.

Also watch: Unsubscribes from the welcome emails, and support questions.

If you are reworking your own onboarding emails, writing a welcome email series covers what each message should do.

Get a fair baseline

A baseline is the normal level of a number before your change. One month is rarely enough. Use three to six months, and note how much the number normally varies. If 60-day retention has ranged between 70 and 78 in every 100 new subscribers, a result of 76 after your change is not evidence of anything. A result of 85 would be.

If your business is seasonal, also compare with the same period a year earlier, so you do not mistake a normal seasonal lift for the effect of your change.

Compare like with like

Where possible, compare groups that differ only in whether they experienced the change:

• Cohorts: compare members who joined just before the change with those who joined just after, at the same point in their membership. Cohort analysis is built for exactly this.
• A hold-out group: for emails and member communications, leave a random portion of new members on the old version for a while, then compare. This is often the most reliable method open to a small site.
• Split tests: for public pages with enough traffic, show both versions at once, as described in A/B testing for membership sites.

Wait long enough, for the right number

Different changes show up at different speeds. A new sales page affects sign-ups within weeks. A change to onboarding affects early retention after one or two months. A new content series or community feature may take several months to show up in churn. Decide in your note when you will judge the result, and do not declare victory or defeat before then.

Also check what else moved. A change that lifts sign-ups but brings in members who leave quickly has not helped. A retention change that lowers churn but floods your inbox with support requests may need refining.

A worked example

The farm newsletter makes the welcome sequence change. Illustrative figures: across the six cohorts before the change, an average of 74 in every 100 new subscribers were still paying after 60 days, ranging from 70 to 78. The first three cohorts after the change came in at 82, 85 and 83.

All three are above the best pre-change cohort, and nothing else notable happened in that period. The owner checks the same months a year earlier and finds no seasonal lift then. That is reasonable evidence the change worked. If only one cohort had beaten 78, or if a large promotion had run at the same time, the honest conclusion would be to keep watching.

A checklist for your next change

1. Write a change note: what, why, main measure, baseline, expected result and when you will judge it.
2. Gather three to six months of baseline data and note the normal range.
3. Avoid making several changes at once in the same area.
4. Use a hold-out group, a cohort comparison or a split test where you can.
5. Record anything else that happens during the measurement period.
6. Judge the result on the date you chose, and write down your conclusion, even if it is inconclusive.