Shelley Endicott
All work A/B testing, a major US telecom brand

Same change, two results

I proposed moving Customer Support out of a dropdown and into the main navigation. On desktop it showed a positive, directional result during the test. On mobile it ran three months and didn't move. The split taught me more than a clean result on both would have.

My role
UX researcher, wrote the hypothesis
Method
Parallel desktop + mobile A/B tests
Length
About 2 months desktop, 3 mobile
Tools
VWO, Looker, Jira

1The problem

On this brand's site, Customer Support sat inside a Support dropdown, one level below the main navigation. People who came to the site needing help had to know to open that menu first.

My hypothesis: support was buried one level too deep. Putting it in the main navigation would get more people who need help to the right page, without hurting sales.

Source: test ticket and tracker. Hypothesis written by me.

2The test

I set it up as two tests running side by side, one on desktop and one on mobile, with the same change in each.

ControlCustomer Support nested in the Support dropdown.
Variant ACustomer Support in the main navigation, relabeled from "Customer Service."
Main metricVisits to the Customer Support page.
Safety checksOrders, revenue, and conversion, so helping support seekers didn't cost sales.

Source: test plan. Traffic split set by the brand team.

3What happened

Desktop

Positive

In the mid-test readout: Customer Support clicks in the nav up 30%, order revenue up 25%, and bounce rate improved. Approved for rollout.

Mobile

Flat

Ran about three months and ended neutral. A directional read of about +13% on orders never became a reportable result.

Mid-test read: desktop figures come from a readout written while the test was still live. Sample sizes and significance weren't recorded anywhere I could access.

Same change, same brand, different results.On desktop the navigation is persistently visible. On mobile it sits behind a menu either way, so moving an item within that menu may have changed much less. My interpretation is that the desktop change affected visibility in a way the mobile change didn't.

Interpretation: my read of the device split. Not tested on its own.

4What the readout couldn't prove

The plan said the main metric was visits to the Customer Support page. The readout reported clicks on the nav item instead. Those answer different questions. Clicks show people found the link. Page visits would show they reached help. The planned metric was never reported.

The desktop numbers were also taken mid-test, and a mid-test number can shift before the test ends. I present them here as exactly that.

Gap: the planned main metric was never reported, and final end-of-test figures weren't available to me.

5What I'd do differently