How Tele2 used A/B tests with AMP to prove revenue impact of mobile load times

July 2019

When it comes to great mobile experiences, speed matters most – for both customer and brand. Tele2, a telecoms provider with strong presence in the Netherlands, set up an A/B test with AMP to measure the impact faster load times would have on their revenue. The kicker? They had no marketing budget behind the test.

In an increasingly mobile-first society, we know that people research and shop on-the-go. They expect websites to load quickly and become frustrated when they don’t. Over half of mobile users (53%) will abandon a mobile site if it takes more than 3 seconds to load1, and 46% of people say that waiting for pages to load is what they dislike most about the mobile web

A recent study revealed that the telecommunications industry is one of the worst performing industries for mobile user experience2 – and with mobile speed being a critical factor to the user’s experience this is a huge opportunity for improvement

So, who better to put mobile speed to the test than one of the Netherland’s fastest-growing telcos3? Tele2 was determined to uncover real impact of mobile load time on their own conversions and revenue. Here’s how they did it.  

Get ready: Identifying the blockers

Before kick-starting their experiment, Tele2 armed themselves with as many insights as possible. Tele2 already recognised site speed as an important part of Conversion Rate Optimisation (CRO). They noticed a steady growth of mobile traffic, while mobile page load times were much higher than on desktop. Across the site, they found that with every additional second of load time, their conversion rate dropped by 3.2%.

Tele2’s developer team already made efforts to optimise images and scripts, update content management systems, and implement loading placeholders, but they knew there was more to be gained. To quickly address the remaining speed blockers, they turned to the web component framework, AMP. This open source, customisable solution provides speed-optimised components, resulting in average page load times under one second. A large part of this speed boost comes from caches that safely and efficiently pre-render AMP pages, while performing additional user-beneficial optimisations.

Get set: As simple as A/B testing

Tele2’s E-Commerce Manager and CRO Specialist teamed up to identify the most popular product page to setup their first test. The developers then built an identical version of the product page in AMP. They performed thorough quality assurance to verify that the two pages looked and functioned identically – this is crucial for setting up a fair test that isolates speed as the only variable. 

At this stage, it was also important to ensure that measurement and analytics were set up properly to allow the CRO team to compare conversions from both pages.

The telco’s digital performance agency iProspect set up an A/B test for this page using Google Ads Drafts & Experiments. To test the impact of overall performance they created a 50-50 traffic split of mobile search towards AMP and non-AMP. 

Go: Getting up-to-speed with the results

The quicker load times were apparent instantly, with the AMP page loading 66% faster than the non-AMP version. And after just a few weeks of testing it was clear that these faster load times had a positive impact on conversion rates.

A full eight weeks of the A/B test showed 56% uplift in click-through rate, guiding more users quickly to the first steps of their purchase. As a result, the e-commerce conversion rate also benefited, with more customers buying from the faster AMP page. 

“The result was spectacular,” explains Lenno de Man, Senior Online Marketing Manager at Tele2. “We A/B tested this setup in Google Ads with no additional marketing budget. We expected the AMP page to increase our paid search performance but the increase in conversion rate was above everyone’s expectations.”

“After analysing the results, it became clear that this was the most successful A/B test Tele2 ever did,” Roderick Eeninkwinkel, Tele2’s Senior E-commerce Manager reveals.

This A/B test clearly demonstrated to Tele2 and iProspect just how much speed can influence conversion rates, and confirms that there is a huge opportunity in the telco industry to improve the user experience by speeding up. Now, the marketing and product teams are expanding the experiment to all traffic sources and investing in further speed improvements. 

Inspired? See how your mobile site measures up with Test My Site and use Page Speed Insights to find out what’s holding you back.

How the mobile masters can help you drive conversions