Email A/B Testing: Subject Lines, Content, Send Times and Open Rate

Email A/B testing means sending two or three versions of the same campaign to a small part of your list, measuring which one performs better, and then sending the winning version to the rest. You can test the subject line, sender name, content and send time, changing only one of them per test.

You use the same method to improve open rate: sender name, subject line, preview text and send time are the four elements that most affect opens, and each can be tested. The most important point in 2026 is that open rate is no longer a reliable metric for most lists. Apple's Mail Privacy Protection distorts open data, so choosing the winner by clicks, conversions or revenue gives you a more accurate result.

What can you test in an email?

Most email marketing tools support four variables. Mailchimp's help documentation, for example, lists subject line, from name, content and send time, and lets you create up to three versions of the chosen variable.

• Subject line: short and direct versus curiosity-driven, personalized versus not, with a number versus without.
• Sender name (from line): the difference between "Brand name" and "First Last, Brand" can be noticeable, especially on B2B lists.
• Content: headline, main image, text length, the wording and position of the call-to-action button, a single product versus a product list.
• Send time: morning, midday and evening sends, or weekdays versus weekends.

Which variable you choose depends on the problem you want to solve. If emails get opened but not clicked, test the content, not the subject line. If they are not opened at all, look first at the open drivers and deliverability below.

What affects email open rate?

In the inbox, the recipient sees three things and decides within seconds whether to open:

• Sender name: a name the recipient recognizes. A sender that changes often, or shows only "info@", lowers trust. Choose your sender name once and keep it stable; if you want to change it, test first.
• Subject line: a phrase that honestly states what the content offers and is short enough not to be cut off on mobile. Stacks of capital letters and excessive exclamation marks hurt both trust and how spam filters treat you.
• Preview text: the short line shown next to the subject line. If you leave it empty, the email client shows the first line of the body, often something meaningless like "view in browser". Write a sentence that completes the subject line.

The fourth element is invisible but decisive: whether the email reaches the inbox at all. According to Gmail's sender guidelines, anyone sending to personal Gmail accounts must set up SPF or DKIM authentication and keep the spam rate reported in Postmaster Tools below 0.3%. Senders of more than 5,000 emails a day also need DMARC and one-click unsubscribe in marketing emails. Testing the subject line of an email that lands in spam gets you nowhere; check this setup first.

List hygiene leads to the same place. Continuing to send to addresses that have not clicked any of your emails for a long time raises the risk of spam complaints. Moving this group into a separate re-engagement flow and removing those who do not respond improves the performance of the rest of the list.

Why is open rate no longer a good way to pick a winner?

When Apple's Mail Privacy Protection is turned on, Apple's support documentation says senders cannot see whether the recipient opened the email, and the recipient's IP address is hidden. The same documentation notes that the protection does not cover links and attachments, which means clicks are still measured.

Mailchimp explains the effect in its own documentation: for recipients with Mail Privacy Protection enabled, emails that reach Apple Mail are reported as opened regardless of what the recipient actually does. In the same document, Mailchimp notes that A/B test results based on open rate may not be accurate and recommends using click rate instead. For the same reason, resending to "non-openers" does not reach these recipients, and Mailchimp suggests using "non-clickers" instead.

The higher the share of Apple Mail users with the protection turned on in your list, the further the reported open rate drifts from real opens. So do not celebrate a rise in open rate before reading it alongside click and conversion data.

The practical result:

• Even in a subject line test, choose the winner by click rate. A good subject line does not just get the open; it gets the open with the right expectation.
• If your online store is connected to the tool, revenue is the most solid metric.
• Do not throw open rate away completely. It still gives you a sense of the trend over time on the same list. But do not make decisions based on a small open-rate difference between two versions.

How large should the test group be?

The test group should be large enough for each version to collect a meaningful number of clicks. What matters here is not the size of your list but the number of clicks you expect.

A simple calculation: on a list of 10,000 people, if you choose a 20% test group and test two versions, each version goes to 1,000 people. If your click rate is around 2%, each version gets about 20 clicks. A difference of 3 or 4 clicks between them is too small to tell apart from chance.

So:

• On small lists, enlarge the test group, or send the test half-and-half to the whole list and carry the learning to the next campaign.
• Test the same question over several campaigns in a row. A result that points the same way in three campaigns is more reliable than one large difference in a single send.
• If you want more detail on sample size and statistical significance, read our article on running A/B tests you can trust.

How long should you wait before choosing the winner?

Tools wait a set period after the test send and then pick the winner automatically. Keeping that wait short is tempting, because you want the campaign to reach the whole list quickly. But clicks spread through the day: an email sent in the morning also gets clicked at lunchtime and in the evening.

A very short wait measures the behavior of early clickers, often the small, most engaged group. If the campaign is not time-sensitive, keep the wait long. If it is time-sensitive (such as a one-day sale), either skip the test or split the whole list and use the result for future campaigns.

What should you watch for in send time tests?

Send time behaves differently from the other variables. Because the versions go out at different times, it is harder to compare results in the same window, and outside factors such as the day of the week, public holidays and payday get mixed in.

• Do not finish a send time test with a single campaign; repeat the same comparison for several weeks.
• If your audience is spread across time zones, check whether the send time is set to each recipient's local time.
• Do not put business and consumer recipients in the same test; the two groups read email differently. Our article on email list segmentation helps you split the list.

Are consent rules different for test sends?

No. Every version of an A/B test is a normal commercial electronic message to the recipient. For recipients in Turkey, Law No. 6563 on the Regulation of Electronic Commerce allows commercial electronic messages only with the recipient's prior consent. Build your test group from opted-in recipients only, and never include anyone who has opted out in any version. If you send to other markets, check the consent rules that apply there.

In short: how do you keep a test log?

If you do not record what each test taught you, the same tests get repeated months later. A simple table is enough:

1. Date and campaign name
2. Variable tested and the versions
3. Hypothesis ("a subject line with a number gets more clicks, because…")
4. Test group size and wait time
5. Winning metric (clicks, conversions or revenue) and the result
6. What will change in the next campaign

If you want campaign planning, segmentation and a testing plan built together, see our email and SMS marketing service.