A/B testing your emails

Every email involves guesses. Will members respond better to a subject line that names the technique or one that names the result? Should the button come first or after the explanation? Is a short email better than a long one for your audience? A/B testing, also called split testing, answers questions like these by sending two versions to comparable groups of subscribers and seeing which one does its job better.

Most email tools make this look easy: write two subject lines, press a button, and the tool declares a winner. The trouble is that the easy version often picks winners by chance and measures the wrong thing. For a membership site with a list in the hundreds or low thousands, a little discipline turns testing from a guessing game into a steady way of learning what your members respond to.

How an email test works

There are two common set-ups:

• Even split: the list is divided randomly in two, each half gets one version and you compare results. This is the most reliable option for a modest list.
• Test then send: a small slice of the list gets each version, the tool waits a set time, picks a winner and sends it to everyone else. It sounds efficient, but with a small list the test slices are tiny and a short waiting time rewards whichever version got the fastest reactions, not the best ones.

Whichever you use, change only one thing between the versions, and make sure both go out at the same time.

Match the measure to what you changed

Different parts of an email do different jobs, so they need different measures:

• Subject line, sender name and preview text affect whether people open. Opens are the obvious measure, but they are unreliable, because some email apps record opens automatically and others hide them. Treat open results as a rough signal and check that clicks move the same way. Remember that the preview line shapes opens too; preview text explains why.
• Call to action wording and placement affect clicks on the main link. Measure clicks on that link specifically, not total clicks, in line with one click that matters.
• Length, format and tone affect whether people act and reply. Measure clicks, replies and, if you can, the action on your site.
• Send time affects when people see the email. Measure clicks over a couple of days rather than the first hour.

Test a question, not a tweak

A single test of two subject lines tells you which of those two lines did better once. That is rarely useful on its own. A better approach is to turn each test into a question about your members, then test it several times with different emails:

• “Do our members respond better when the subject names a specific technique rather than a general benefit?”
• “Do members click more when the button appears before the explanation?”
• “Do short, personal emails get more replies than longer ones?”

If the same answer comes up across three or four different emails, you have learned something you can use every time you write.

Worked example: a subject-line question over four sends

Imagine a hypothetical calligraphy membership run by Mei, with about 1,200 members on her list. Her question is whether naming the specific technique beats naming the benefit. The results below are illustrative:

1. Send one. A: “Smoother curves in your lowercase letters.” B: “The two-pressure stroke for lowercase curves.” Main-link clicks: A 34, B 41.
2. Send two. A: “Finally, even spacing.” B: “Spacing letters with the n-width rule.” Clicks: A 29, B 38.
3. Send three. A: “Make your flourishes look effortless.” B: “Three entry strokes for cleaner flourishes.” Clicks: A 45, B 44.
4. Send four. A: “Better envelopes this month.” B: “Addressing envelopes with a slant guide.” Clicks: A 31, B 40.

No single send is conclusive; differences of a handful of clicks on a list this size can easily be chance. But the technique-named version won three times out of four and tied once. Mei can reasonably adopt it as her default while staying open to changing her mind.

Test your automated emails too

Welcome emails, event reminders and renewal notices are sent again and again, which makes them ideal for testing. Instead of one send, run the split for a month or two until each version has reached a good number of people. Change one email in a sequence at a time, and keep the rest identical so you know what made the difference. Improvements here keep paying off with every new member.

Keep a simple test log

Tests are only valuable if you remember them. Keep a running document with one line per test:

• Date and email
• The question being tested
• What differed between the versions
• Audience and size of each group
• The measure you chose in advance
• Results, and whether they were clear or too close to call
• What you decided

Over time the log becomes a style guide written by your own members.

Run your first email test

1. Write down one question about your members that affects many emails.
2. Choose the element to change and the measure that fits it.
3. Use an even random split and send both versions at the same time.
4. Wait at least a couple of days before reading the results.
5. Repeat the same question on at least three different emails.
6. Record every result in your log, including the inconclusive ones.
7. Adopt the answer only when it holds up across several sends.