How the Best Tech Companies Run Growth Experiments

A critical part of growth in product-led companies is their approach to experiments.

It’s the experiments that often unlock the door to opportunities you wouldn’t have otherwise discovered. They help you to understand your users better and give you insights on how to create more value for them.

We’ve gotten advice from Patreon, Pinterest, SurveyMonkey, Invision and more on how to make growth experiments a roaring success in your company.

Here are five great pieces of advice you can use to improve your approach to growth experiments.

1. A hypothesis is not a prediction

The hypothesis is a central part of your experiment doc – it’s a statement you believe to be true about your users.

A common mistake when forming your hypothesis is to state it as a prediction based on a metric you think will improve, something that fails to articulate what you believe to be true of your users.

Let’s take an example of a growth team who are trying to improve the checkout flow of an online e-commerce store and compare two common hypothesis types.

Prediction-based hypothesis:

User-based hypothesis:

The problem with basing a hypothesis on a prediction formed around a metric is that you haven’t articulated what you believe to be true about your users. By not doing this you won’t be clear on what learning the experiment has provided you about those users.

It’s difficult to articulate what you’ve learned from an experiment if you don’t have a clear hypothesis.

2. An experiment helps you understand the potential upside of an initiative

Nearly all experiment docs will contain a section for the predicted upside from an experiment. If this is successful, how will it impact one of your core business metrics?

The reality is if you could accurately predict the upside of a potential change then you wouldn’t need to run the experiment in the first place.

Experiments are another form of research. They help you to better understand the potential success or failure of an initiative and how it could impact your metrics if successful.

3. Your ability to properly scope an experiment will have a significant impact on the success or failure of your growth team.

To minimize your risk from taking on experiments, continuously look for ways to cut down on the scope of that experiment.

If you can keep the scope of your experiments low:

      • It’s easier to get buy-in from stakeholders because you’re reducing the potential downside of the experiment failing.
      • You won’t be so dependent on a small number of experiments succeeding as you’ll be able to run a higher volume of experiments that require a smaller amount of work.

However, your scope also needs to be meaningful enough that if your experiment fails, you’re happy to move on. This is where a lot of growth teams make mistakes.

If you continually look to reduce the scope of your experiments, it can stop you from pursuing initiatives that have both higher rewards and risks. Minimize the scope of the experiment too much and what you end up executing on might not be sufficient enough to prove or disprove an idea is worth investing in.

You need to find a balance between minimizing the amount of work you put into the test while understanding the level of investment required to understand if your hypothesis is right or wrong. That means if the experiment fails, you’re not left wondering:

“What if we had made the user experience a little better? Was the experience we put live a fair enough representation of the end product that it gave us enough data to make a final decision on this initiative?”

But how do you manage experiments that are a higher risk because they need a considerable upfront investment?

Adopting this approach means, at the very worst if the big initiative doesn’t work, there is still a lot of potential for success with the other things you’re doing. That’s the best way to manage risk, with diversification.

That’s how you can decrease the cost of taking big swings.

4. Autonomy is the best way to scale experiment ideas

The best way to scale ideas across your growth team is to create an environment where no one person owns the ideation process.

At Pinterest, one of the fastest growing tech companies on the planet, they work to empower individuals to not only come up with different ideas for experiments but to also be responsible for taking that idea from concept to execution.

They’ve found this approach has helped them to get a better diversity of experiments along with better quality of work as the person who has the original idea is also responsible for executing on that idea.

Every two to three weeks they have a meeting called the ‘Experiment Idea Review.’ Everyone attends that meeting – engineers, PM’s and designers. Each person who is submitting an idea needs to complete a template based on Jeff’s hype framework.

Once the person receives feedback from the community they’re free to pursue that idea whether the feedback was positive or negative.

SurveyMonkey has also taken this approach to experiments. They restructured their team to democratize growth across the company.

They still keep a centralized team to make decisions around best practices as well as the prioritization of tests, but providing the company with these tools has allowed them to develop a culture of experimentation at scale.

5. Experiments are a great way to introduce the company to growth

The purpose of experiments is ultimately to learn something. You have a clear hypothesis about something and running experiments can help you understand if you’re right or wrong.

Like any other form of research, what you learn can benefit people across different functions within your company. Growth teams can also learn a lot by talking with other groups in the company (e.g., sales, support, services).

So how can you use experiments to get more people in your company involved in growth?

Use the above advice from the experts in growth to turn experiments into high impact wins for your company in 2019.

And remember the GrowthTLDR podcast aims to provide you with the best expert advice for growing your product-led funnel.

VP of Marketing

Kieran Flanagan is the VP of ​Marketing at HubSpot. He is responsible for the growth of HubSpot’s freemium products, monetization of their freemium funnels and optimization of their global web strategy.
You might also like ...
The Assessment Every Business Needs to Do ASAP

Unless you’re Amazon, Instacart or Zoom, you need to assess your customers’ engagement with your product. Sam Richard shares the dos and don’ts of customer health scoring.

by Sam Richard
Enterprise Growth Wisdom From IBM’s Zoltan Szalas

The secret to his success? A data-driven approach to growth—and a genuine love for all things data.

by Sam Richard