A/B testing sends two to five versions of the same newsletter, called variations, to find the one that works best. Each has its own content, subject line, sender and preview text (the short line shown after the subject in the inbox). MailerPress measures opens and clicks and identifies the best variation.
This feature requires MailerPress Pro, version 2.1.0 or later of both MailerPress and MailerPress Pro. Without MailerPress Pro, Campaigns > All A/B campaigns only shows a presentation of the feature.
Choose a test method
- Test first, then send winner (named Test, then send the winner in the sending panel): a sample, meaning a part of your audience, receives the variations first. After the test window, MailerPress selects the best variation, which can then be sent to the rest of the audience. This is the default.
- Send all variations now (named Split the whole audience in the sending panel): the whole audience is split between the variations right away. There is no winner to wait for. You compare the results after the send.
Change only one element between variations (for example the subject line or the main call to action), so you know what made the difference.
Create an A/B testing campaign
- Go to MailerPress > Campaigns, then click All A/B campaigns in the Experiments section of the sidebar.
- Click Create Campaign. The Create A/B Testing Campaign assistant opens.
- Editor: choose Visual Editor or Raw HTML.
- Basic Info: enter a Campaign name and a Subject, and select at least one list. You can also select tags (labels you put on contacts).
- A/B Test: choose the method and set the options described below.
- Template (visual editor only): start from scratch, from a template or from a previous campaign. All variations start with this content.
- Review: check the summary, then click Create Campaign. MailerPress creates one editable campaign per variation and opens variation A in the editor.
A/B test options
| Option | Description |
|---|---|
| Number of variations | From 2 to 5 variations, labelled A to E. |
| Sample split / Audience split | The share of recipients each variation receives: inside the test sample with the winner method, across the whole audience with the split method. |
| Test sample percentage | Winner method only. The part of the audience that receives the test send. Default 20%, maximum 90%. |
| Test duration and Unit | Winner method only. How long MailerPress waits before picking the winner, in minutes, hours or days. Default 24 hours, maximum 30 days. |
| Fallback variation if there is no data | Winner method only. The variation sent to the remaining audience if no opens or clicks are recorded during the test. |
| After the test window | Winner method only. Select the winner and wait for manual confirmation (you send the winner yourself) or Automatically send the winner to the remaining audience. |
Convert an existing draft into an A/B test
- Open a draft classic campaign in the email editor.
- Open the Settings menu in the top toolbar, choose Convert to A/B test and confirm.
Your content becomes variation A and is copied to a new variation B. No email is sent. The test uses the winner method with a 20% sample and a 24-hour test window. You can change these settings before launching.
Edit the variations
Edit each variation like a normal campaign. Use the Variations button in the editor toolbar to switch between them. When you open an empty variation, MailerPress copies the content of the variation you were editing, so you only change what you want to test.
If AI is configured in MailerPress Pro, variations B and following show a Generate with AI button. It rewrites the text and call to action but keeps the design, colors, images, layout and footer. Review the result before sending.
To go back to a single campaign while the test is still a draft, open the Variations menu, choose Convert to standard campaign and select the Variation to keep. The other variations and the A/B test are deleted.
Launch the test
When your variations are ready, click Preview & Send in the editor. The A/B panel has four steps:
- Content: pick a Variation to edit and set its From sender, Subject line and Preview text. Each variation has its own values. You can send a test of the selected variation from here.
- Audience: under Send to, select lists and tags, or a segment (a group of contacts that match rules you define). MailerPress shows the number of recipients selected.
- Method: confirm the method. The Who receives what bar shows how the audience is split. With Test, then send the winner, set the Test group size (1 to 90%), Pick the winner after and Send the winner automatically. Under Advanced settings, you can change the Split of the test group and the variation used If there is no data. With Split the whole audience, set the Share per variation.
- Launch: review the summary. Click Launch A/B test to start now, or turn on Schedule for later, pick a date and click Schedule A/B test.
Before launch, every variation must have saved content, and the audience must contain at least as many contacts as there are variations. Once a test is launched or scheduled, its setup can no longer be changed. To keep your settings without launching, click Save draft.
You can also launch a draft test with its saved setup: use the Launch row action in All A/B campaigns, or the Launch A/B test button in All campaigns.
How the winner is chosen
MailerPress gives each variation a score: 60% of its click rate (share of recipients who clicked a link) plus 40% of its open rate (share of recipients who opened the email). If two variations have the same score, MailerPress compares the click rate, then the number of clicks, the open rate and the number of opens.
With the winner method, MailerPress evaluates the test at the end of the test window. If no opens or clicks were recorded, it uses the fallback variation. Then:
- If the winner is sent automatically, MailerPress sends it to the contacts of the audience who did not receive a variation.
- Otherwise, the test status becomes Winner selected. Open All A/B campaigns and click Send winner on the test to send it to the remaining audience.
With the split method, there is no winner to send: everyone already received a variation.
Follow the results
Campaigns > All A/B campaigns lists your tests with their method, a performance overview showing the leading variation, and their status: Draft, Scheduled, Test in progress, Winner selected, Completed, Canceled or Error.
Click View statistics in the row actions to open the full report:
- a verdict naming the best variation and explaining why it performs better;
- Compare the variations: a table ranked from best to worst with each variation’s emails sent, open rate, click rate, score and email preview;
- an Email click heatmap: use Variation to display to see where recipients clicked in each variation.
When each variation went to few contacts, MailerPress warns you that the results are a trend, not a certainty.
The Refresh metrics row action recalculates the scores and records the current best variation. Use it only after the test window has ended: before that, it can select a winner too early, and sends it right away if the winner is sent automatically.
Cancel or delete a test
- Cancel test is available for scheduled and running tests. A scheduled test goes back to draft. For a running test, the pending winner evaluation and the remaining scheduled sends are stopped. Emails already sent cannot be recalled.
- Deleting an A/B testing campaign also deletes its variations. A running test cannot be deleted: cancel it or wait until it is complete.
For the statistics of your other campaigns, see Campaign statistics and reports.