Testing Methodology

Split-Testing for SEO purposes has some similarities with traditional CRO focused split-testing, and some big differences. The primary difference, which can be difficult to grasp initially, is that CRO testing buckets users into two groups, whereas SEO testing buckets pages into two groups.

Test Setup

With CRO focused testing, user A and user B may see different versions of the same page to one another. For example, user A may see a tabbed layout (the control) and user B may see a flat layout (the variant) of a product page; if user A refreshes the page they will continue to see the tabbed layout (as they have been bucketed as control).

With SEO focused testing, user A and user B will see the version of any page as on another, as will search bots and any other visitors. However, different pages in the same family of pages (e.g. product pages, article pages, or region pages) will appear differently; for example the blue widget page may have a tabbed layout and the green widget page a flat layout. These pages use identical templates and similar content to one another. It is important to recognise that there still only exists one copy of each page - it will either be a control of a variant, but not both.

In this fashion a group of pages, such as product pages, will be split into two variations which will have some change that we are looking to test. Search bots, including Googlebot, will crawl the pages and see the same thing as users would see for each page.

Measuring Impact

There will now be two types of pages in the search index, and each type of page may differ in terms of organic search performance. The variant group of pages now may begin to perform better than the control group of pages, which would result in (relatively) more organic search traffic to those pages.

The DistilledODN platform connects to your analytics via an API and records the organic search traffic to both the control and variant pages. It can then do an analysis of which group of pages are performing better and show you the expected impact were you to roll that change out across all of your pages.

The mathematics of the analysis system account for other factors that may change. Things such as:

  • Google Algorithm Updates
  • Seasonality
  • Competitor's changes
  • Unrelated changes to the site

Are all controlled for by the platform, using the control pages to change the expected 'baseline' behaviour for the variant pages. Some of the work used is based on the paper Inferring causal impact using Bayesian structural time series, which was published by Google.

Get a demo

If you're interested in a short demo, please fill in this form and one of the ODN team will ping you an email.

Alternatively, if you have any other questions, feel free to drop us a line at contact@distilledodn.com or check out our FAQ page and deck.