How an AI sidecar product drove 30% of sign-ups: Eraser’s founder on building and growing DiagramGPT

Editor’s Note: This article, by Eraser founder Shin Kim, first appeared in Kyle Poyar’s newsletter, Growth Unhinged, which explores the unexpected behind today’s fastest-growing startups. You can subscribe to get the latest from Kyle here.

Eraser — a documents and diagrams tool for engineering teams that I founded in November 2020 — has grown rapidly, with sign-ups scaling by 20x in the last year.

We focus on serving developers with targeted features like diagram-as-code and GitHub syncing being central to the product. And we have a track record of trying less-explored growth strategies such as virtual office partnerships and an ungated product experience.

In May 2023, we released a sidecar product called DiagramGPT, an AI tool that can generate diagrams from natural language or code snippets. I wrote this post because I wanted to share our experience with other software teams that have built internal LLM-powered prototypes but are struggling to figure out next steps. It’s a case study reflecting on what our team did and learned, and ultimately why your team may also want to build an AI sidecar product.

According to OpenView’s forthcoming 2023 SaaS benchmarks report, 46% of SaaS companies launched AI features over the last year and another 31% are actively building or testing AI features. In other words, more than 3 out of 4 SaaS companies have had or currently have AI R&D initiatives.

However, the bar to integrate anything into an existing mature product is usually very high. Two resulting failure modes are:

• The AI prototype loses steam going through endless iterations
• A significantly scoped down version gets shipped to test the waters but fails to generate excitement from customers

As we show in the example of DiagramGPT, there is another path forward. A sidecar product can be a mechanism to deliver a bold, undiluted vision of an AI feature to users quickly. It will generate qualitative and quantitative feedback that can help bring the AI feature into the main product. And of course, if it grows legs, the sidecar product can become a reliable organic user acquisition loop. DiagramGPT accounts for 30% of sign-ups for Eraser today.

Building DiagramGPT

When we got our hands on GPT-4 like the rest of the world in the Spring 2023, it was soon clear that LLMs were finally smart enough to generate diagrams. We were able to input natural language or code snippets and output beautiful diagrams. The early results were compelling enough that I decided to fly to Boston for a week to collaborate with our Founding Engineer, Yoel Tadmor, and kick off a new AI project.

On the first day, we discussed whether the generative diagramming functionality should be a product feature or its own sidecar product. We quickly settled on the sidecar product because we did not want to burden it with the craftsmanship that goes into building features in our core product. We wanted to find a footing in the quickly shifting LLM landscape, and the best way would be to ship something as soon as possible.

Early mockup of DiagramGPT, drawn using Eraser

Despite focusing on speed, we still made sure to create moments of delight for the user. For example, instead of just showing a spinner while the diagram is generated, we created a streaming experience where interim outputs are rendered frame by frame. This made the waiting less dull for the user and the product experience more dynamic.

Streamed diagram generation makes the waiting fun!

Another explicit decision was to create a separate brand identity called DiagramGPT for the sidecar product. By not calling it Eraser AI, we gave the sidecar product enough distance from our main brand such that we’d have no problem moving on from it if it flopped. We also built DiagramGPT in dark mode despite our then light mode website and even crafted a slightly different logo.

DiagramGPT with a different brand identity than the rest of Eraser

Launching DiagramGPT

We treated launching DiagramGPT like a product launch. We put a date on the calendar and coordinated engineering and marketing prep against that date. We invited friendly users to play around with it in advance and recorded a scripted demo video. As the launch neared, we had enough internal conviction in DiagramGPT that we knew that we wanted to spend some more time polishing the UX. As a result, we ended up pushing back the launch a few days to improve the streaming experience and diagram layouts.

One of the final decisions was what the call-to-action (CTA) would be. DiagramGPT was an un-gated sidecar product that lived on the marketing website. If DiagramGPT was successful, what business impact did we want it to drive?

The obvious option would be to nudge users to create an Eraser account. However, that didn’t feel right because the bigger opportunity seemed to be in user discovery as we explored the nascent problem space of AI diagramming. Learning first, and growth second. So instead, we made “Let’s chat” / “Request early access” the main CTA.

On launch day, we posted our demo video to all social channels. It was well received on Twitter, especially considering our small following of <1k followers. Leveraging that social proof, we pitched DiagramGPT to the growing number of popular AI newsletters. Several large newsletters picked up DiagramGPT on their next issue, which ended up creating a bigger impact than the original social posts.

Our Twitter post created the initial splash but the newsletter features had a bigger impact

After the Launch

We were glad to have gone with call scheduling as the CTA as we conducted over 40 user discovery calls in the following month scheduled from DiagramGPT. Not every call was packed with insights of course. But, it allowed us to survey a broad cross-section of users interested in AI diagramming ranging from PhD students to software architects at large enterprises. The conversations also gave us conviction to build and ship an API version of DiagramGPT.

DiagramGPT also provided usage data with a large enough sample size that allowed us to build a better AI diagramming feature into the core product. For example, we observed that about half of the generation attempts were actually re-tries or slight tweaks of the original prompts. This told us that iterating on diagrams with “AI edits” would be important and made sure to ship that feature in our core product.

The AI diagram feature that ultimately was shipped to the core Eraser product

DiagramGPT ultimately delivered on sign-ups as well. After the initial spike from the launch and newsletter features, it wasn’t immediately obvious that the incoming traffic would be sustained. But to our surprise, traffic actually steadily grew mainly via word of mouth. Unlike text or image generation, diagram generation is a less crowded space. Simply being the first with a compelling demo was actually enough to draw sustained interest.

In order to encourage visitors to sign up for Eraser, we gated diagram editing behind a sign-up wall. You’d be able to generate a diagram, but not edit it without signing up. Since AI generations are rarely perfect as is, we observed that serious users with work use cases would often sign up.

DiagramGPT grew to eventually account for ~30% of Eraser sign-ups. As usage scaled, so did cost to serve and we ultimately introduced a limit on how many diagrams could be generated before creating an Eraser account.

DiagramGPT gradually compounding sign-ups

Reasons to build an AI sidecar product

#1. Create a sustainable and organic user acquisition channel.

If successful, your sidecar can become a reliable, zero-to-low-cost user acquisition channel that compounds on its own. It’s a way to place your shiniest feature outside of the sign-up wall or paywall so that even low intent users can experience your product with minimal friction.

In our case, the launch marketing and the newsletter mentions seeded initial interest in DiagramGPT. Afterwards, it grew via word-of-mouth without much additional effort from our team. The growing public interest to find and adopt AI tools likely worked in our favor. Paid marketing for the sidecar product did not make sense for us and likely doesn’t for most companies unless conversion is highly efficient.

#2. Validate demand for AI features.

Your AI feature could either be a game changer or a massive waste of time and it’s hard to know which it will be without trying. AI sidecar products are a way to quickly test demand without committing to invasive changes to the core product. You can run bold experiments because they are easy to discard if they don’t wok.

#3. Leverage usage data for product development.

Building AI features today is hard in part because there aren’t established design patterns to rely on. For example, should your specialized AI feature have a chat interface or a non-chat interface? Referencing peers or competitors won’t help because they are likely still figuring it out too. However, with sufficient traction, and if you have time to wait, the sidecar product can generate usage data that will steer your decisions.

Reasons not to build an AI sidecar product

While our experience at Eraser has been positive, an AI sidecar product isn’t for everyone. A few areas to consider.

#1. It could be a source of distraction.

Unlike other growth projects, it will require real engineering and design time, probably equivalent to building a medium-sized feature. Would you rather ship a core product feature or ship the sidecar product? That said, AI sidecar products are often more lightweight since most software teams have already spent the time in the last 12 months (see OpenView survey data mentioned above) to build internal R&D prototypes. For those teams, the marginal investment to ship the sidecar will be minimal.

#2. It could cannibalize usage of your core product.

If you give too much value away in the un-gated sidecar product, users will have less reason to sign up for your core product. In Eraser’s case, if DiagramGPT were generating perfect diagrams every single time, we could end up with a lot of DiagramGPT users who don’t bother to sign up for Eraser. The ideal sidecar product experience should deliver concrete value but leave the user craving for more.

#3. LLM usage costs could balloon depending on your LLM stack (OSS vs. OpenAI, GPT-3.5 vs. GPT-4, etc.).

This is a gotcha since software teams usually don’t need to think about the infra cost of a marketing site. But it will be worthwhile to confirm that the unit economics pencil out for your AI sidecar product. Does your sign-up conversion, paid conversion, and LTV justify the LLM infra cost involved?