Most e-commerce brands and marketers have an opinion about what makes a good email. They “think” they know which subject lines will produce the best open rates, which offers will convert, and which layouts will perform best. Some of those opinions are right. Some are wrong. And the only way to know the difference is to test and see what will perform best for your audience.
A/B testing, also called split testing, is the practice of sending two versions of the same email to different portions of your audience to see which one performs better. It removes guesswork from your email marketing program and replaces it with evidence. Done consistently, A/B testing is one of the most reliable ways to improve your email performance over time without changing anything about your products, your email list, or your sending frequency.
Klaviyo has robust A/B testing built into both campaigns and flows. Here is how to use A/B testing properly, and where to focus your testing attention first.
How A/B Testing Works in Klaviyo
For campaigns, Klaviyo’s A/B testing is straightforward to set up. When you create a campaign, you will see an option to add a variant. You can test up to three variants of a campaign, with each variant receiving a defined percentage of your audience during the testing window. At the end of the testing window, Klaviyo automatically sends the winning variant to the remainder of your email list based on whichever metric you define as the winner. Examples of winning metrics or KPIs for A/B testing includes: open rate for subject line tests, click rate for content tests.
For flows, A/B testing works at the individual email level. Inside the flow builder, you can add a split to any email action and create variants with different content, subject lines, or timing. Flow A/B tests run continuously, splitting traffic between variants over time and accumulating data with every subscriber who enters the flow.
A/B Testing: What Do I Test First?
Not everything is worth testing, and testing too many things at once makes it impossible to know what drove a result. The best approach for A/B testing in email marketing is to start with the variables that have the highest potential impact on your most important metrics and test them one at a time.
Subject Lines
Subject lines have the highest impact on open rate, and open rate is the first gate your email has to pass through. A subject line test that lifts your open rate by five percentage points means meaningfully more subscribers seeing your content on every future send.
Subject line tests are also the fastest to get clean results on because you can use open rate as the winning A/B testing metric, and open rates are measurable quickly. Run your test for two to four hours on a portion of your list (typically 20 percent split equally between the two variants), then send the winner to the remaining 80 percent.
When A/B testing subject lines, change only one variable at a time. If you are testing whether a question performs better than a statement, make sure the rest of the subject line is as similar as possible. If you change the length and the format and the offer all at once, you will not know which change drove the result.
A/B Testing: Email Send Time
The time that you send an email affects who sees it. Subscribers who are at their desks at 9am may respond differently than subscribers who are scrolling their phones at 8pm. The right send time depends on your specific audience, their habits, and the nature of your email content.
To test send time, keep everything else constant (same subject line, same content, same audience) and split your test between two different send windows. Common tests include morning versus evening, Tuesday versus Thursday, or weekday versus weekend.
Klaviyo also has a Smart Send Time feature that analyzes your audience’s past engagement patterns and recommends optimal send times based on that data. This is worth testing against your intuition. Klaviyo also now can send emails based on when your specific subscriber is most likely to open the email. Mind blown!
A/B Testing: Email Copy
Testing the body copy in your emails is particularly valuable for your highest-volume flows, where even small improvements in conversion rate compound over hundreds or thousands of subscribers over time.
Copy tests might include different lengths (shorter and more direct versus longer and more detailed), different angles (emotional storytelling versus rational product features), or different levels of formality. Keep your subject line and design constant when testing copy so that any difference in results can be attributed to the copy change.
A/B Testing: Calls to Action
Your call-to-action button is what drives clicks, and the language you use on it has a measurable impact. Testing button copy is one of the lower-effort, higher-impact tests available to you because a single word change on a button can meaningfully shift click rate.
Common CTA tests include action-oriented versus passive language (“Shop the Collection” versus “Explore our Collection”), benefit-focused versus action-focused (“Get 20% Off” vs. “Apply Your Discount”), and first-person versus second-person framing (“Claim My Offer” vs. “Claim Your Offer”).
A/B Testing: Offer and Incentive
Testing different incentive types or amounts can reveal significant revenue insights. If you are currently offering a flat 10 percent discount for your e-commerce store, testing it against free shipping or a dollar-amount discount can tell you which motivates your specific audience more effectively.
Offer tests need longer windows and larger sample sizes to be statistically meaningful because you are measuring conversion rate rather than open rate or click rate. Conversion events are less frequent, so you need more data to reach a confident conclusion. Give offer tests at least a week, ideally two, and make sure each variant reaches at least 500 to 1,000 recipients.
A/B Testing: Email Design and Layout
Design tests can reveal whether a more minimal template outperforms a visually rich one, whether product images at the top versus the bottom of an email affect click rate, or whether single-column versus multi-column layouts perform differently with your audience.
Design tests are worth running on your most important flows, particularly Welcome Series and Abandoned Cart flows, where the template will be seen by every new subscriber or every person who abandons a cart. A layout improvement that lifts conversion rate by one percent in your abandoned cart flow generates compounding revenue with every abandoned cart that flows through it.
A/B Testing: How Long Should I Run a Test?
The most common mistake in A/B testing is ending a test too early. If you check your results after a couple hours and one variant is ahead, it can be tempting to call the winner. Please do not do this. Early data in A/B testing is noisy, and a variant that looks like a winner at hour one may not hold up when more data comes in.
For campaign tests, the standard approach is a test window of two to four hours for subject line tests (because open rate data accumulates quickly) and 24 to 48 hours for click rate or conversion rate tests. For flow tests, let variants run until each has received at least 250 to 500 recipients before drawing conclusions.
Statistical significance matters. Klaviyo shows you confidence levels in your test results. A result with 95 percent statistical significance means you can be reasonably confident the result is not due to chance. If your test shows one variant ahead but at lower confidence, more data is needed before you declare a winner.
Building a Testing Culture
The brands that consistently outperform in email marketing are not the ones that ran one test and implemented the winner. They are the ones that test something in every send, keep a log of what they tested and what they found, and use those findings to continuously raise the baseline performance of their program.
Building a simple test log is one of the most practical habits you can adopt. After each test, record what you tested, what the variants were, which won, and by how much. Over time, this log becomes a picture of what works for your specific audience, and it removes a lot of the guesswork from future email decisions.
If you want help setting up a structured testing program in Klaviyo or identifying the highest-impact tests to run first in your account, we are happy to help.
Continue reading: Subject line formulas that improve open rates (with Klaviyo examples) and Understanding your Klaviyo dashboard: The metrics that actually matter.