Product

How the Best Tech Companies Run Growth Experiments

May 8, 2020

Editor’s note: This post was updated on May 8, 2020.

A critical part of growth in product-led companies is their approach to experiments.

It’s the experiments that often unlock the door to opportunities you wouldn’t have otherwise discovered. They help you to understand your users better and give you insights on how to create more value for them.

Related: Product Led Growth 101

So, we wanted to hear from the experts. We asked leaders at Patreon, Pinterest, SurveyMonkey, InVision and more how they make growth experiments a roaring success at their organizations—and how the rest of us can do the same.

Here are five great pieces of advice you can use to improve your approach to growth experiments.

1. A hypothesis is not a prediction

The hypothesis is a central part of your experiment doc—it’s a statement you believe to be true about your users.

A common mistake when forming your hypothesis is to state it as a prediction based on a metric you think will improve, something that fails to articulate what you believe to be true of your users.

Let’s use an example of a growth team who’s trying to improve the checkout flow of an online e-commerce store and compare two common hypothesis types.

Prediction-based hypothesis:

User-based hypothesis:

The problem with basing a hypothesis on a prediction formed around a metric is that you haven’t articulated what you believe to be true about your users. Without doing this, you won’t be clear on what information the experiment provided you about those users.

It’s difficult to articulate what you’ve learned from an experiment if you don’t have a clear hypothesis.

2. An experiment helps you understand the potential upside of an initiative

Nearly all experiment docs contain a section for the predicted upside from an experiment. If this is successful, how will it impact one of your core business metrics?

The reality is this: If you could accurately predict the upside of a potential change, you wouldn’t need to run the experiment in the first place.

Experiments are another form of research. They help you better understand the potential success or failure of an initiative and how it could impact your metrics if successful.

3. Your ability to properly scope an experiment will have a significant impact on the success or failure of your growth team

To minimize your risk from taking on experiments, continuously look for ways to cut down on the scope of that experiment.

If you can keep the scope of your experiments low:

  • It’s easier to get buy-in from stakeholders because you’re reducing the potential downside of the experiment failing.
  • You won’t be so dependent on a small number of experiments succeeding as you’ll be able to run a higher volume of experiments that require a smaller amount of work.

But your scope also needs to be meaningful enough that if your experiment fails, you’re happy to move on.

This is where a lot of growth teams make mistakes.

If you continually look to reduce the scope of your experiments, it can stop you from pursuing initiatives that have both higher rewards and risks. Minimize the scope of the experiment too much, and what you end up executing on might not be sufficient enough to prove or disprove that an idea is worth the investment.

Find a balance between minimizing the amount of work you put into the test while understanding the level of investment required to understand whether your hypothesis is right or wrong.

That means if the experiment fails, you’re not left wondering:

“What if we had made the user experience a little better? Was the experience a fair enough representation of the end product such that it gave us enough data to make a final decision on this initiative?”

But how do you manage experiments that are a higher risk because they need a considerable upfront investment?

Adopting this approach means that if the big initiative doesn’t work, there’s still a lot of potential for success with the other things you’re doing. That’s the best way to manage risk—with diversification.

That’s how you can decrease the cost of taking big swings.

4. Autonomy is the best way to scale experiment ideas

The best way to scale ideas across your growth team is to create an environment where no one person owns the ideation process.

At Pinterest, they empower individuals to not only come up with different ideas for experiments but to also be responsible for taking that idea from concept to execution.

They’ve found this approach has helped them to get a better diversity of experiments along with better quality of work, as the person who has the original idea is also responsible for executing on that idea.

Every two to three weeks they have a meeting called the Experiment Idea Review. Everyone attends that meeting—engineers, PMs and designers. Each person who is submitting an idea needs to complete a template.

Once the person receives feedback from the community, they’re free to pursue that idea even if the feedback was negative.

SurveyMonkey also takes this approach with experiments. They restructured their team to democratize growth across the company.

They still keep a centralized team to make decisions around best practices as well as the prioritization of tests, but providing the company with these tools allowed them to develop a culture of experimentation at scale.

5. Experiments are a great way to introduce the company to growth

The purpose of experiments is ultimately to learn something. You have a clear hypothesis about something, and running experiments can help you understand if you’re right or wrong.

Related: Building a System for Growth

Like any other form of research, what you learn can benefit people across different functions within your company. Growth teams can also learn a lot by talking with other groups in the company (e.g., sales, support, services).

So how can you use experiments to get more people in your company involved in growth?

Use the above advice from the experts in growth to turn experiments into high-impact wins for your company.

For more expert advice for growing your product-led funnel, check out my podcast, GrowthTLDR.

KieranFlanagan

VP of Marketing

Kieran Flanagan is the VP of ​Marketing at HubSpot. He is responsible for the growth of HubSpot’s freemium products, monetization of their freemium funnels and optimization of their global web strategy.