UX experiments / Craft & speed
Craft & speed · A/B test · replicated
Short or long ad titles: Bing's search ads experiment
Search ads: keep the short title, or pull text from the first line up into a longer title?
Make your call. Which version won, and why? Decide before you open the result. Guessing first is what trains your judgment.
Made your call? See the result
Revenue up 12%, over $100M a year in the US.
What happened
The experiment was replicated several times over a long period. Key user-experience metrics did not drop significantly.
Why it worked
More information in the line people actually read first. The change was trivial to build and had been sitting in the backlog.
The takeaway
Cheap, boring ideas can be the biggest wins. Test the backlog, not only the roadmap.
Source
- Kohavi & Thomke · The Surprising Power of Online Experiments (HBR 2017) (exp-platform.com)
More experiments to call
Practice the call
Call experiments like these every day.
Irreplaceable gives you a few real experiments a day to call before you see the result, and tracks how your judgment improves. Start with a free baseline: 12 decisions, about 12 minutes.