How the Best Tech Companies Run Growth Experiments

A critical part of growth in product-led companies is their approach to experiments.

It’s the experiments that often unlock the door to opportunities you wouldn’t have otherwise discovered. They help you to understand your users better and give you insights on how to create more value for them.

We’ve gotten advice from Patreon, Pinterest, SurveyMonkey, Invision and more on how to make growth experiments a roaring success in your company.

Here are five great pieces of advice you can use to improve your approach to growth experiments.

1. A hypothesis is not a prediction

The hypothesis is a central part of your experiment doc – it’s a statement you believe to be true about your users.

A common mistake when forming your hypothesis is to state it as a prediction based on a metric you think will improve, something that fails to articulate what you believe to be true of your users.

Let’s take an example of a growth team who are trying to improve the checkout flow of an online e-commerce store and compare two common hypothesis types.

Prediction-based hypothesis:

User-based hypothesis:

The problem with basing a hypothesis on a prediction formed around a metric is that you haven’t articulated what you believe to be true about your users. By not doing this you won’t be clear on what learning the experiment has provided you about those users.

It’s difficult to articulate what you’ve learned from an experiment if you don’t have a clear hypothesis.

2. An experiment helps you understand the potential upside of an initiative

Nearly all experiment docs will contain a section for the predicted upside from an experiment. If this is successful, how will it impact one of your core business metrics?

The reality is if you could accurately predict the upside of a potential change then you wouldn’t need to run the experiment in the first place.

Experiments are another form of research. They help you to better understand the potential success or failure of an initiative and how it could impact your metrics if successful.

3. Your ability to properly scope an experiment will have a significant impact on the success or failure of your growth team.

To minimize your risk from taking on experiments, continuously look for ways to cut down on the scope of that experiment.

If you can keep the scope of your experiments low:

      • It’s easier to get buy-in from stakeholders because you’re reducing the potential downside of the experiment failing.
      • You won’t be so dependent on a small number of experiments succeeding as you’ll be able to run a higher volume of experiments that require a smaller amount of work.

However, your scope also needs to be meaningful enough that if your experiment fails, you’re happy to move on. This is where a lot of growth teams make mistakes.

If you continually look to reduce the scope of your experiments, it can stop you from pursuing initiatives that have both higher rewards and risks. Minimize the scope of the experiment too much and what you end up executing on might not be sufficient enough to prove or disprove an idea is worth investing in.

You need to find a balance between minimizing the amount of work you put into the test while understanding the level of investment required to understand if your hypothesis is right or wrong. That means if the experiment fails, you’re not left wondering:

“What if we had made the user experience a little better? Was the experience we put live a fair enough representation of the end product that it gave us enough data to make a final decision on this initiative?”

But how do you manage experiments that are a higher risk because they need a considerable upfront investment?

Adopting this approach means, at the very worst if the big initiative doesn’t work, there is still a lot of potential for success with the other things you’re doing. That’s the best way to manage risk, with diversification.

That’s how you can decrease the cost of taking big swings.

4. Autonomy is the best way to scale experiment ideas

The best way to scale ideas across your growth team is to create an environment where no one person owns the ideation process.

At Pinterest, one of the fastest growing tech companies on the planet, they work to empower individuals to not only come up with different ideas for experiments but to also be responsible for taking that idea from concept to execution.

They’ve found this approach has helped them to get a better diversity of experiments along with better quality of work as the person who has the original idea is also responsible for executing on that idea.

Every two to three weeks they have a meeting called the ‘Experiment Idea Review.’ Everyone attends that meeting – engineers, PM’s and designers. Each person who is submitting an idea needs to complete a template based on Jeff’s hype framework.

Once the person receives feedback from the community they’re free to pursue that idea whether the feedback was positive or negative.

SurveyMonkey has also taken this approach to experiments. They restructured their team to democratize growth across the company.

They still keep a centralized team to make decisions around best practices as well as the prioritization of tests, but providing the company with these tools has allowed them to develop a culture of experimentation at scale.

5. Experiments are a great way to introduce the company to growth

The purpose of experiments is ultimately to learn something. You have a clear hypothesis about something and running experiments can help you understand if you’re right or wrong.

Like any other form of research, what you learn can benefit people across different functions within your company. Growth teams can also learn a lot by talking with other groups in the company (e.g., sales, support, services).

So how can you use experiments to get more people in your company involved in growth?

Use the above advice from the experts in growth to turn experiments into high impact wins for your company in 2019.

And remember the GrowthTLDR podcast aims to provide you with the best expert advice for growing your product-led funnel.

You might also like ...
How Calendly Serves 5,000,000 People on a Monthly Basis [Podcast]

On the last episode of the season, Tope Awotona, Founder & CEO of Calendly, shares his take on product led growth and what finally convinced him the was the right business to build.

by Mackey Craven
Webflow’s Mission to Empower as Many People as Possible [Podcast]

This episode features Vlad Magdalin, Co-founder & CEO of Webflow, a business that took four tries to get off the ground. Find out how they think about creating a product that’s applicable to as many people as possible and why adding incremental value is more important than shipping a perfect product.

by Mackey Craven
There’s an App for That: A Guide to the Product Management Stack

Many tools have emerged to help PMs do their jobs more effectively. These make up the product stack, composed of more than a dozen categories. But what is the actual function of each category? Here, we drill down.

by Shannon Bauman