The graphic above captures a stark reality: regular product validation is essential to mitigate the risk of startup failure. With constant feedback, a startup’s trajectory can stay on its intended path. Lean experiments act as checkpoints, realigning the startup’s course with the market’s pulse.
Let’s say you’re a chef experimenting with a new recipe to see what your customers like best. You tweak the ingredients slightly each time and keep making the original recipe. The original recipe acts as a “control group” — it’s the standard you compare the new versions against. In tech and startup experiments, this control group is essential to understand the actual effect of your changes.TikTok’s Algorithm Tuning
TikTok’s approach to refining its content recommendation algorithm is quite similar. Imagine TikTok’s algorithm as a recipe they’re trying to perfect. Their goal is to make their app so engaging that users spend more time on it and enjoy the experience. To achieve this, they didn’t change the entire recipe at once. Instead, they made small changes — like a chef altering one ingredient at a time.
For instance, TikTok might tweak its algorithm to show users more videos that people typically watch all the way through, thinking this might keep users engaged longer. But how do they know if this change is working? Here’s where the control group comes in. While they test this new algorithm version on some users, they keep the original version for others. These others — the control group — help TikTok see what user behavior would have been like without the changes.
By comparing the behavior of users experiencing the new algorithm with those using the original (the control group), TikTok can see if their new ‘recipe’ is more effective.
They look at how long users stay on the app, how much they interact with the content, and whether they keep returning. If the new algorithm shows better results than the control group, it’s a win. If not, it’s back to the drawing board, like our chef returning to the kitchen to try another ingredient tweak.
Making small, measured changes and comparing them to a control group is at the heart of lean experimentation in the tech and startup world. It’s about learning what works best by trying out new ideas on a small scale before deciding if they’re good enough to implement fully.Airbnb’s Photographic Transformation
Airbnb is a notable example of a startup that used experimentation to improve its product and scale its business. In their early days, Airbnb was struggling to increase bookings. The team hypothesized that the quality of the property images was impacting users’ booking decisions. To test this hypothesis, they experimented: they went to New York, rented a professional camera, and replaced the amateur photographs of listings with high-quality photos.
The results were immediate and significant. The professional photos doubled the weekly revenue to $400 per week in New York. This experiment was so successful that Airbnb rolled out professional photography as a standard service feature.Running Experiments at Large-Scale
This lean experiment allowed Airbnb to identify a simple yet critical lever for their platform’s success and scale it efficiently. The company continued to use a data-driven approach to decision-making, constantly running experiments to optimize every aspect of its service, from search algorithms to user interface design.“Feedback is the breakfast of champions.” — Ken Blanchard
The Airbnb and the TikTok cases are often cited in discussions on lean startup methodologies and growth hacking, showing the impact that intelligent, well-executed experiments can have on a company’s trajectory.
This underscores a crucial point: experimentation isn’t just beneficial; it’s a competitive edge. For startups, especially in the early stages, time is the most precious currency. Wasting it isn’t an option. Establishing a pipeline of experiments is imperative to ensure continuous learning and improvement. In my work with founders, I emphasize the critical role of structured experimentation — it’s often the deciding factor in outpacing the competition and achieving success.Here are two examples of potential experiments:Experiment 1: Landing Page Conversions
Experiment 2: User Retention
- KPI to Influence: Landing Page Conversion Rates
- Hypothesis: I believe that adding a testimonial section to our landing page will enhance users’ trust and increase conversion rates.
- Experiment Design: Introducing a testimonial section on the landing page will improve conversion rates. Create two versions of the landing page — Version A without testimonials and Version B with a section showcasing customer testimonials.
- Timing: Run for two weeks to allow users to engage with the new element.
- Measurement: Measure and compare conversion rates between the two landing page versions.
Watch Out For Correlation vs. Causation
- KPI to Influence: User Retention Rates
- Hypothesis: I believe that offering an in-app tutorial will elucidate the app’s value, leading to higher user retention within the first week of use.
- Experiment Design: Implementing an in-app tutorial will increase user retention within the first seven days.. Group A receives no tutorial, while Group B users are introduced to a step-by-step in-app tutorial.
- Timing: Monitor user behavior over a month to assess retention over different milestones (Day 1, Day 7, Day 30).
- Measurement: Compare the retention rates of both groups at each milestone.
Understanding correlation and causation is vital if you plan to conduct lean experiments. Correlation suggests a relationship between two variables but doesn’t prove that one causes the other. Causation, on the other hand, means that one event results from the occurrence of the other event; there is a cause-and-effect relationship.
For instance, a startup may see that as their social media engagement goes up, so do their sales. They might conclude that increasing social media efforts leads directly to sales (causation). However, sales and social media engagement are increasing due to a third factor, such as a seasonal trend or a market-wide uptick in demand (correlation).
Misinterpreting these two concepts can lead startups astray. Suppose a startup ramps up social media spending, assuming it will increase sales. In that case, they might be investing resources into an area with no direct effect on revenue, potentially neglecting other areas that could substantially impact their growth. This is why designing experiments that can isolate variables and test causative relationships, not just correlations, is crucial.Sample Size
The sample size in any experiment is crucial to make data-driven decisions. A sample size that’s too small can lead to misleading results, where the success or failure of an experiment is merely due to chance rather than a true reflection of your market’s response.
For instance, if a startup testing a new feature only solicits feedback from a handful of users, they might wrongly assume the feature is a hit or miss based on this limited input. This could lead them to fully roll out a feature that hasn’t been adequately vetted or to discard a potentially valuable one prematurely.Here are some actionable insights on how to calculate the sample size for your startup’s experiments:
What About Early Stage Startups?
- Set the Confidence Level: Typically, a confidence level of 95% is used, which means you can be 95% certain that the results fall within a specific range.
- Determine the Margin of Error: Decide on the maximum margin of error you are willing to accept. A smaller margin of error requires a larger sample size.
- Estimate the Variability: If you have historical data, use it to estimate your metric's variability (standard deviation). If not, you might assume a 50% variability as a conservative estimate.
- Use a Sample Size Calculator: Utilize an online sample size calculator where you can input the above parameters (confidence level, margin of error, and variability) to estimate the sample size needed.
- Adjust for Population Size: If your customer base is small, you must adjust your sample size to reflect the population size.
- Plan for Response Rate: Consider the response rate if you conduct surveys. Not everyone will respond, so you may need to send out more invitations than the number in your sample size.
At CodeCheck, we achieved an early product channel fit with millions of users. This advantage could have been leveraged for running experiments to refine our approach even before we nailed down the product-market fit. Looking back, that knowledge would have been invaluable. Yet, it’s essential to acknowledge that such circumstances are outliers, not the standard experience for most startups.
Traditional experimental options like A/B testing might be constrained in most cases due to a small user base. Anyway, startups in that phase are one big experiment with a singular focus: attaining product-market fit.
Demand testing via landing pages is a practical experiment for B2C startups or B2B ventures with larger markets. An MVP test, too, can take on an experimental nature by inviting distinct cohorts for closed trials, ensuring there’s always a benchmark group for comparison.
However, it’s essential to recognize the challenge of running frequent experiments without a substantial user pool. The key isn’t just to experiment but to do so smartly, ensuring that every test yields valuable insights.
Introducing the Experiment Canvas
To streamline the experimentation process, I’ve crafted the Experiment Canvas, a strategic tool designed to bring order to the experimentation process. This canvas is a roadmap for startups to navigate through the process of testing and learning.