Set up an A/B test
Open the newsletter you want to test.
In Test multiple versions of this newsletter, choose what to test from What to test, then click Add versions.
Click Edit on each version to make your changes. To add more, click Add version. You can add up to 26 versions.
When you're ready, schedule the newsletter. In Test on, enter the percentage of your list that gets the test.
That percentage is split evenly across your versions. For example, testing 2 versions on 40% of your list sends each version to 20%. The other 60% gets the winner.
In Pick the winner by, choose the metric. Subject line tests always use unique open rate.
In Wait before picking a winner, choose how long the test runs, from 4 hours to 1 week.
Review the summary, then click Schedule delivery. Once sending begins, it can't be stopped.
How the winner is picked
When the wait time ends, Seguno compares the versions on the metric you chose and sends the winner to the rest of your list automatically. There's no minimum number of opens or clicks. The winner is picked at the time you set, so a longer wait gives subscribers more time to respond and makes the result more reliable.
Metric | What it measures |
Unique open rate | The share of recipients who opened the version at least once. Opens from Apple Mail Privacy Protection are left out, because that feature can open emails automatically and make open rates look higher than they are. |
Unique click rate | The share of recipients who clicked a link in the version at least once. |
Revenue per recipient* | The revenue from a version divided by the number of people who received it. |
*Revenue usually takes longer to come in than opens or clicks. If you pick the winner by revenue per recipient, consider a wait of at least 1 day.
Limits and beta details
Versions: up to 26 per newsletter.
Test size: up to 100% of your list. At 100%, every subscriber gets one of the versions and no one is left to receive a winner, so the test works as a head-to-head comparison.
Small tests: with many versions or a small list, each version reaches fewer subscribers, and results are less reliable.
Where it works: newsletters only. Automations aren't supported yet.
Beta: available to a limited group of merchants and fully supported. Some screens and features may change during the beta.
Frequently Asked Questions
Why is my test's open rate different from my other reports? A/B tests leave out opens from Apple Mail Privacy Protection, which can open emails automatically. Your test open rates will often be lower than open rates elsewhere, but they better reflect who actually opened.
When does the rest of my list get the newsletter? After the wait time ends. If you schedule for 8:00 am with a 4-hour wait, the winner goes out around 12:00 pm. With a 1-week wait, the rest of your list gets it a week later, so keep waits short for time-sensitive promotions.
Can I pick the winner myself? No. The winner is picked automatically by the metric you chose at the end of the wait time.
Where do I see my results? In the newsletter's report, broken out by version.
What happens if the tests tie? In the unlikely event of a tie, the first variant will be selected as the winner to ensure the full delivery is sent.
Can I edit a version after scheduling? No. Make sure every version is final before you click Schedule delivery.
How big should my test be? There's no minimum, but each version needs enough recipients for the result to mean something. If your list is small, test fewer versions or test on a larger share of your list.
Will a subscriber get more than one version? No. Each subscriber gets one email: either a test version or the winner.
Do discounts and excluded segments work the same way? Yes. Your discount and any excluded segments apply to every version and to the winner send.
Can I A/B test automations? Not yet. A/B testing works with newsletters only for now.