Experimentation Services

Experimentation is a powerful approach used during the optimisation of a product to validate and evaluate changes in alignment with specific objectives. It allows us to validate changes with real users, using structured tests to measure impact on behaviour, usage, and business outcomes. Rather than relying on assumptions or intuition alone, experimentation puts changes into the real world and measures how they perform under live conditions.

This approach is particularly valuable when a product has a strong existing user base and established patterns of behaviour. It creates an opportunity to iterate and optimise based on evidence, not guesswork, helping teams make confident decisions that support key metrics like conversion, engagement, retention, or revenue.

For example, if we want to improve sign-ups or usage of a new feature, we can test changes that increase its visibility, improve its usability, or clarify its value. If a variant outperforms the original, we know the change is working. If not, we dig into the data to understand why and test it again.

Successful experimentation is more than just running A/B tests. It’s a structured process that begins with insight and ends with confident decisions. We guide teams through each of the following key steps:

Start with data

Every experiment should begin with a clear understanding of the problem. We review existing analytics and user research to identify patterns, friction points or underperforming areas, or conduct our own research to gather new qualitative or quantitative insights. For example, data might show that users aren’t clicking a feature, abandoning a form, or struggling with a navigation flow.

Formulate a hypothesis

We define a hypothesis using a clear format:
“We believe that if we do X, Y will happen. We’ll know this is true if we see an increase in Z.”
This gives the experiment purpose and provides a clear way to assess success. For instance:
“We believe that if we simplify the onboarding flow for new course participants, more users will complete the setup process. We’ll know this is true if the onboarding completion rate increases.”“We believe that if we move the CTA to the top of the landing page, more users will click it. We’ll know this is true if the click-through rate increases.”

Define success metrics

Once the hypothesis is clear, we identify what to measure. This might include click rates, sign-ups, usage of a specific feature, or time to task completion. Metrics should directly reflect user behaviour and be relevant to business goals.

Design and run the experiment

Depending on the situation, we might use A/B testing, multivariate testing, or split user flows. Participants are exposed to different design variations, and we collect quantitative data to see which version performs best. Sometimes we also use moderated usability testing to understand the why behind observed behaviours.

Analyse the results

We examine the data to determine whether the change had a statistically significant impact. If the hypothesis is confirmed, we recommend implementation. If not, we review the findings, adjust the hypothesis, and continue testing. Even “failed” tests provide useful insight and direction to further shape strategy going forward.

Implement and iterate

Validated changes are rolled out more broadly, but experimentation doesn’t stop there. With new data in hand, we return to the beginning of the cycle, identifying fresh opportunities for improvement and running new experiments to keep growing the product.

Experimentation works best when there is a steady flow of traffic or users, as larger sample sizes provide more reliable results. For enterprise-level or high-traffic products, this creates the opportunity for continuous learning and improvement over time.

We often run tests in areas such as:

• Landing page layout and messaging
• Placement and design of CTAs
• Onboarding flows and sign-up processes
• Feature discoverability and learnability
• Pricing pages or plan comparison views

Each test is designed around specific goals, whether that’s improving activation rates, increasing usage of a key feature, or uncovering why users are dropping off.

Typical outcomes include:

• Clear evidence of which changes improve user behaviour, such as increased clicks, sign-ups or feature use
• A better understanding of what works, what doesn’t, and why, helping teams to focus their efforts more effectively
• Design improvements that support key goals like conversion, engagement or retention
• Reduced risk when rolling out new features or interface changes, by testing them at a smaller scale first
• A growing library of learnings over time, helping inform future decisions and avoid repeating past mistakes