Email A/B Split Testing

A/B Split Testing allows a user to configure two unique Mailings and determine which was more effective. Enrollment Manager allows you to A/B Test on nearly any property of a Mailing. The only exceptions that must match in both mailings are the Outreach List, the Campaign, the Subscription and in the Advanced section the Delay list synchronization check box. Other than those 4 items anything and everything else can be A/B tested. Best practices would be to keep the mailings fairly similar and A/B test one thing at a time so you know why one mailing did better than the other but we don’t force that restriction on you.

For example, you might have two identical emails but try using different Subject lines. Or, try using one text based message and one HTML Rich message. A/B testing allows you to determine which will be more effective before sending the bulk of your messages.

You start by configuring two unique mailings, making sure to match up the fields mentioned earlier. Then you'll click A/B Tests/New to create your new test. In here you will select a name for the A/B Test, select the two Mailings you created earlier the Winning Condition, Test Duration and the Split Percentage.

Winning Condition determines which criteria to use to select your winning message (number of unique opens or number of unique clicks).

Test Duration determines how long the test will last (1-24 hours), which also indicates when the winning message will be automatically sent.

Split Percentage indicates the percent of the list that will included in each of the three mailings. You can select anywhere from 10% to 40%. The image is set to 20% so on a list with 100 members, 20 recipients would receive Mailing A, 20 recipients would receive Mailing B and when the test was completed the remaining 60 recipients would receive the Winning Mailing.

Please note that the Winning Mailing field is read only and will be populated when the test completes and we determine and create the Winning Mailing.

Once you’re happy with the way your A/B Test is set up and click the Save button the Testing and Schedule sections will become available to you. 

Testing
The Testing section works nearly identically to its Mailing counterpart. You enter an e-mail address and click the Send Test button.

The only difference between the A/B Test version of Send Test Email is that you will not get a separated HTML and TEXT version like you do when sending a test from a Mailing. You will instead get a copy of Mailing A and a copy of Mailing B exactly how the recipient would see it.

Scheduling
After you’ve tested both Mailings and verified that everything is ready to be sent you move on to the Schedule section. Like the Testing section, the Schedule section is nearly identical to its mailing counterpart.

Once you enter a date and click Send Later or click on Send Immediately you will be prompted with the same Accept Policy page that displays when you have selected to schedule the A/B Test as well as the recipient counts for the Marketing List and Additional Lists. (You must have a total of at least 50 recipients to send an A/B Test).

Once your A/B Test is scheduled you will see Mailing A and Mailing B start to Synchronize and go out as if you had scheduled them the typical way.

Results
Once the test is complete there will be another section called the Snapshot section. This section will display the Opens, Implied Opens, Total Opens and Clicks for both Mailing A and Mailing B at the time we picked a winner. This is to provide a record of why the winning mailing was selected. It’s possible that Mailing A might be ahead after 1 hour when your test ends however when you check it 3 hours later Mailing B now seems like it should have won. You can always reference this Snapshot section to see who was winning when the test completed.

After the winner is chosen we will update the A/B Test with a link to the Winning Mailing.
Comments