Should You Test That? When to Engage in SEO Split Tests

Posted by Portent

This blog was written by Tim Mehta, a onetime Conversion Rate Optimization Strategist with Portent, Inc .

Running A/ B/ n experimentations( aka “Split Tests”) to improve your search engine standings has been in the SEO toolkit for longer than countless would think. Moz actually published an essay back in 2015 broaching the subject, which is a great summary of how you can run these assessments.

What I want to cover here is understanding the best time to run an SEO split-test, and not how you should be running them.

I run a CRO program at an agency that’s well-known for SEO. The SEO team brings me in when they are preparing to run an SEO split-test to ensure we are following best practises when it comes to experimentation. This has given me the chance to see how SEOs are currently approaching split-testing, and where we can improve upon the process.

One of my greatest remarks when working on these projects has been the most pressing and often forgot question: “Should we research that? ”

Risks of racing useless SEO split-tests

Below you will find a few potential risks of leading an SEO split-test. You might be willing to make some of these risks, while there are others you will most definitely want to avoid.

Wasted resources

With on-page split-tests( not SEO split-tests ), you can be much more agile and propel multiple evaluations per month without expending significant resources. Plus, the pre-test and post-test evaluations are much easier to perform with the calculators and formulas readily accessible through our tools.

With SEO split-testing, there’s a ponderous amount of lifting that goes into planning a test out, actually defining it up, and then executing it.

What you’re essentially doing is taking an existing template of same sheets on your locate and separating it up into two( or more) separate templates. This asks substantial blooming resources and constitutes more danger, as you can’t simply “turn the test off” if things aren’t going well. As you probably know, formerly you’ve made a change to hurt your positions, it’s a lengthy uphill battle to get them back.

The pre-test analysis to anticipate how long you need to run the test to reach statistical significance is more complex and takes up a good deal of hour with SEO split-testing. It’s not as simple as, “Which one does more organic commerce? ” because each variance you research has peculiar aspects to it. For precedent, if you choose to split-test the product page template of half of your makes versus the other half of them, the actual concoctions in each change can play a part in its performance.

Therefore, you have to create a projection of organic traffic for each variation based on the sheets that exist within it, and then compare the actual data to your estimations. Inherently, exploiting your estimate as your main indicator of failure or success is dangerous, because a projection is just an improved guess and not necessarily what reality indicates.

For the post-test analysis, since you’re measuring organic traffic versus a hypothesized jutting, you have to look at other data points to determine success. Evan Hall, Senior SEO Strategist at Portent, asks 😛 TAGEND

“Always use establishing data. Look at relevant keyword rankings, keyword sounds, and CTR( if you rely Google Search Console ). You can safely rely on GSC data if you’ve received it pairs your Google Analytics numbers pretty well.”

The time to plan a test, be developed further on your live locate, “end” the test( if it is necessary to ), and analyze the test after the fact are all demanding tasks.

Because of this, you need to make sure you’re feed experiments with a strong hypothesis and fairly variations in the modification versus the original that you will see a significant difference in conduct from them. You also need to corroborate the data that would point to success, as the organic congestion versus your projection alone isn’t reliable enough to be confident in your results.

Unable to flake the results

There are many parts that go into your search engine rankings that are out of your hands. These lead to a robust number of outside variables that can impact your test results and lead to false positives, or false negatives.

This hurts your ability to learn from the test: was it our variation’s template or another outside factor that led to the results? Regrettably, with Google and other search engines, there’s never a definite channel to answer that question.

Without validation and understanding that it was the exact alters you fixed that led to the results, you won’t be able to scale the prevailing concept to other canals or specific areas of the site. Although, if you are focused more on individual outcomes and not learnings, then this might not be as much of increased risk for you.

When to run an SEO split-test Uncertainty around keyword or query performance

If your series of sheets for a particular category have a wide variety of keywords/ queries that users search for when looking for that topic, you can safely engage in a meta name or meta description SEO split-test.

From a conversion proportion perspective, having a more relevant keyword in relation to a user’s intent will generally lead to higher date. Although, as mentioned, the majority of members of your exams won’t be winners.

For example, we have a client in the tire retail industry who shown in in the SERPs for all kinds of “tire” queries. This includes things like winter tires, seasonal tires, carry-on tires, etc. We hypothesized that including the more specific phrase “winter” tires instead of “tires” in our meta names in the winter months would lead to a higher CTR and more organic traffic from the SERPs. While our results intent up being inconclusive, we learned that changing this meta title did not hurt organic congestion or CTR, which gives us a prime the possibilities for a follow-up test.

You can also implement this tactic to experiment out a higher-volume keyword in your metadata. But this approach is also never a sure thing, and is worth testing first. As highlighted in this Whiteboard Friday from Moz, they appreciated “up to 20 -plus-percent lowerings in organic freight after revising meta message in titles and so forth to target the more commonly-searched-for variant.”

In other paroles, targeting higher-volume keywords seems like a no-brainer, but it’s always worth testing first.

Proof of abstraction and gamble mitigation for large-scale locates

This is the most common call for running an SEO split-test. Therefore, we reached out to some experts to get their take on when this scenario turns into a prime opportunity for testing.

Jenny Halasz, President at JLH Marketing, talks about use SEO split-tests to prove out notions or minds that haven’t gotten buy-in:

“What I have found many times is that suggesting to a consumer they try something on a smaller subset of sheets or categories as a’ proof of concept’ is extremely effective. By keeping a command and focusing on veers rather than whole numbers, I can often show a patron how deepening a template has a positive impact on search and/ or conversions.”

She goes on to reference an existing example that stresses an alternate testing tactic other than manipulating templates 😛 TAGEND

“I’m in the middle of a test right now with a client to see if some smart-alecky internal relate within a subset of commodities( squandering InLinks and OnCrawl’s InRank) will work for them. This measure is really fun to watch because the change is not really a template reform, but a piloting mutate within a category. If it acts as I expect it to, it could mean a entire redesign for this client.”

Ian Laurie emphasizes the purpose of applying SEO split-testing as a risk mitigation tool. He explains:

“For me, it’s about flake. If you’re going to implement a conversion impacting tens or the thousands of sheets, it pays to run a split measure. Google’s unpredictable, and changing that numerous sheets can have a big up- or downside. By testing, you can manage risk and get purchaser( external or internal) buy-in on organization sites.”

If you’re responsible for a large site that is heavily dependent on non-branded organic explorations, it pays to test before releasing the amendment to your templates, irrespective of the size of the reform. In this case, you aren’t necessarily are waiting for a “winner.” Your desire should be “does not transgres anything.”

Evan Hall emphasizes that you can utilize split-testing as an instrument for excuse smaller mutates that you’re having trouble going buy-in for 😛 TAGEND

“Budget justification is for testing deepens that require a lot of developer hours or writing. Some e-commerce sites may was necessary to set a blurb of textbook on every PLP, but that might require a lot of writing for something not guaranteed to work. If the test suggests that content will provide 1.5% more organic traffic, then the effort of writing all that text is justifiable.”

Determine big changes to your templates

In experimentation, there’s a metric called a “Minimum Detectable Effect”( MDE ). This metric represents the percentage gap in achievement you expect the variation to have versus the original. The more the modifications and more differences between your original and your alteration, the higher your MDE should be.

The graph below emphasized the need for the lower your MDE( filch ), the more traffic you will need to reach a statistically significant result. In turn, the highest the MDE( lift ), the less sample size you will need.

For example, If you are redesigning the place architecture of your commodity sheet templates, you should consider making it noticeably different from both a visual and back-end( system formation) perspective. While used experiment or on-page A/ B testing may have led to the brand-new design or designing, it’s still unsure whether the proposed modifications will affect rankings.

This should be the most common reason that you run an SEO split test. Given all of the subjectivity of the pre-test and post-test analysis, you want to make sure your variation relents a different enough arise to be confident that the modification did in fact have significant implications. Of route, with bigger modifications, comes bigger risks.

While larger sites have the comfort of testing smaller things, they are always at the benevolence of their own guesswork. For less robust areas, if you are going to run an SEO split test on a template, it needs to be different fairly not only for users to behave differently but for Google to evaluate and rank your sheet differently as well.

Communicating experimentation for SEO split-tests

Regardless of your SEO expertise, communicating with stakeholders about experimentation requires a skill set of its own.

The possibilities with testing are highly volatile. Some parties expect every experiment to be a winner. Some expect you to give them definitive answers on what will work better. Unfortunately, these are false possibilities. To avoid them, you need to establish realistic hopes early on for your manager, client, or whoever “youre running” a separate experiment for.

Expectation 1: Most of your tests will neglect

This understanding is a pillar of all successful experimentation curricula. For beings not close to the subject, it’s also the hardest pill to immerse. You have to get them to accept the fact that the time and effort that goes into the first iteration of a test will most likely lead to an inconclusive or losing exam.

The most valuable aspect of experimentation and split-testing is the iterative process each evaluation experiences. The genuine outcome of successful experimentation, regardless if it’s SEO split-testing or other types, is the culmination of multiple experiments that is conducive to gradual expanded in major KPIs.

Expectation 2: You are working with probabilities , not sure thing

This hope applies especially to SEO split-testing, as you are utilizing a variety of metrics as indirect signals of success. This helps people understand that, even though they are you reach 99% important, there are no guarantees of the results once the triumph discrepancy is implemented.

This principle likewise gives you wiggle-room for pre-test and post-test analysis. That doesn’t mean you can manipulate the data in your praise, but does mean you don’t need to spend hours and hours coming up with an empirically data-driven projection. It too allows you to utilize your subjective expert opinion based on all the metrics you are analyzing to determine success.

Expectation 3: You need a large enough sample size

Without a large enough sample size, you shouldn’t even entertain the idea of passing an SEO split test unless your stakeholders are patient enough to wait several months for outcomes.

Sam Nenzer, a consultant for SearchPilot and Distilled, explains how to know if you have enough traffic for testing: “Over the course of our experience with SEO split testing, we’ve rendered a rule of thumb: if a site slouse of similar pages doesn’t receive at least 1,000 organic periods per day in total, it’s going to be very hard to measure any uplift from your separate test.”

Therefore, if your locate doesn’t have the right traffic, you may want to default to low-risk implementations or competitive investigate to ratify your ideas.

Possibility 4: The aim of experimentation is to mitigate risk with the potential of rendition betterment

The key period here is “potential” performance improvement. If your assessment crops a triumphing variance, and you implement it across your place, don’t expect the same upshots to happen as you ensure during the test. The true-blue destination for all researching is to introduce new ideas to your place with very low risk and potential for improved metrics.

For example, if you are upgrading from the structure or system of a PDP template to accommodate a Google algorithm change, the goal isn’t inevitably to increase organic congestion. The goal is to reduce the negative impact you may see from the algorithm convert.

Let your stakeholders know that you can also utilize split-testing to improve business value or internal efficiencies. This includes things like secreting code updates that users never receive, or a URL/ CMS inform for an organization of sheets or several microsites at a time.


While it’s tempting to run an SEO split test, it’s vital that you understand the intrinsic hazards of it to ensure that you’re coming the true value you need out of it. This will help inform you on when the situation calls for a split evaluation or another approach. You likewise need to be communicating experimentation with realistic expectancies from the get-go.

There are major inherent dangers of hiring with SEO split-testing that you don’t interpret with on-page research that CRO typically fees, including squandered resources and non scalable results.

Some of the scenarios where you should feel confident in pursue with an SEO split test include where you’re uncertain of keyword and inquiry carry-on, proof-of-concept and risk mitigation for larger-scale websites, reasons for plans that require robust resources, and when you’re considering offsetting big changes to your templates.

And remember, one of the biggest challenges of experimentation is properly communicating it to others. Everyone has different anticipations for testing, so you need to get ahead of it and address those expectations right away.

If there are other scenarios for or gambles are connected with SEO split-testing that you’ve seen in your own work, delight share in the comments below.

Sign up for The Moz Top 10, a semimonthly mailer informing you on the top ten hottest parts of SEO news, tip-off, and rad relates uncovered by the Moz team. Think of it as your exclusive accept of nonsense you don’t have time to hunt down but want to read!

Read more: