When it comes to growth and conversion rate optimization, there's one topic that's been a hot potato lately: Experimentation. Experimentation is the secret sauce that takes products, services, or even entire businesses to the next level. But what does this process look like at scale? Let me share the Experimentation Loop process I've created to make sure testing is integrated into organization and it isn't only about running one a/b test every now and then. Ready? Let's dive in!
The entire process embodies a looped nature, underlining the iterative aspect of experimentation. Each cycle concludes, integrates its learnings, and embarks on a new, more informed round of experimentation. Let's zoom in on each step:
Initiatives Aligned with company direction: Before even starting the experimentation process, it's crucial to ensure all ideas and initiatives are aligned with the company's strategy and direction. This step sets the stage to ensure all subsequent actions align with broader business objectives, and define metrics you want to move.
Ideas Input: After alignment with the roadmap, the process moves to gathering inputs for generating experiment ideas from multiple sources:
Data and UX review: Delving into available data and user experience insights.
User research or feedback: Sourcing information directly through user research or through feedback channels.
Competitive research: Analyzing direct, indirect competitors, and companies you look up to.
Past experiments data: Meta-analysis of test run by your company or from publicly available data - eg. https://goodui.org/blog/
Hypothesis Prioritization: Once ideas are collected, they are prioritized, determining which hypotheses to test first.
Research and Analysis: Ahead of the actual solution design, conduct thorough research and analysis to better inform your designs and potential outcome.
Solution Ideation: For the shortlisted hypotheses, generate potential solutions and interventions.
Experiment Design: Detail out the structure of the experiment, deciding the metrics to be tracked, outlining data collection methods, segments included, product areas where the test will be conducted, and calculating test duration for each intervention.
Intervention Prioritization: Rank the interventions or changes based on criteria eg. from priotization frameworks to determine which ones to test initially.
Experiment Execution: Conduct the experiment for a designated period, often ranging from 1 to 4 weeks.
Post-Experiment Analysis: If the assumptions of the experiment are validated, the following steps are taken:
Analyze WHY the effect was observed: Analyze the results, different segments, and user behaviors to create WHYPHOTHESIS
Report the test: Conclude the test with report of what has been observed and share with your org
Making Code Production-Ready: Refine the tested changes or features for a full-scale roll-out, while keeping the winning variant on production.
Deploy Changes: Introduce the changes to a larger user base.
Decision on doubling down on this bet: If we explored something new that's interesting for us, at this stage we should decide if we wanna exploit the effect we observed. If the assumptions of the experiment are NOT validated, the following steps are taken:
Analyze WHY the effect was observed: Analyze the results, different segments, and user behaviors to create WHYPHOTHESIS
Report the test: Conclude the test with report of what has been observed and share with your org
Decision if the Experiment should be Redesigned?: If the experiment didn't hit the expected marks, consider if it's viable to retest the hypothesis with a different approach or intervention.
Go Back to Solution Ideation or check where the company wants to go next: This way we close the loop and make sure our testing cycle doesn't loose the momentum.
Notes from Kuba:
I have not included Stakeholder Communication in this process but it's critical to this process success and need to be happening constantly, but within intervals adapted to your org.
On top you should collaborate with different teams on ideation, solution design, getting them aligned on what you're going to test, educating them on the findings from your tests, and keeping their ideas in the experimentation backlog, so that you can manage expectations ;)
Do you need help starting and scaling experimentation process in your company?