Mastering API Analytics for API Programs: Cohort Retention Analysis
There are few metrics more critical than retention for a platform business. If you’re acquiring customers for $25, but they stop using your API after a month, then you have a leaky boat. Don’t spend more money on developer acquisition until retention is fixed. This requires accurate measurement of API retention.
If you came from a web or mobile product background, you may already be familiar with mobile retention to measure how many acquired users keep using a mobile app. Growing a B2B platform requires tracking similar KPIs to measure the success of your acquisition and product strategies. This article will dig into the best practices for tracking and increasing API retention.
What is API retention?
Retention measures the percent of users within a cohort that return and stay active with your product. What is considered active for your product depends on the type of product. For a streaming mobile app, being active on a day may mean playing a song. For a payments API, it could be processing a credit card payment on a day.
To measure retention accurately, you need to segment your users into cohorts. Usually, this is done by sign up date, but for APIs it’s recommended to segment based on integration date or Time of First Hello World. This is the date when a user first integrated and made their first transaction through your API. Then, every day, week, or month later we count the number of unique users who returned and performed an action which you consider a strong indicator for being active.
The below chart shows new users segmented by the day of their first API call. From there, we track the percent of users that are still active on our API each day after. This chart shows 37.5% of new users are still actively making API calls the first day after integration. Five days after integration, only 24% of our initial cohort was still active.
What is good API retention?
While only 24% of a cohort is active 5 days after integration doesn’t seem high, this chart can be quite good depending on your use case. Retention should not measure engagement or stickiness levels. Rather, you should use retention curves to see how many users are retained. The best way to tell is how flat the curve is.
A product with a retention curve that continues to slope down until hitting 0% has poor retention. Whereas, a product with a retention curve that levels out or increases slightly has good retention.
Day 1 should have the largest dropoff. Most platforms have a set of developers that sign up, play around with the API in a sandbox, but never returns. Once a customer finds value, they keep consuming the API on Day 2, Day 3, and so on. These users found value in the API and stay integrated. On the other hand, if your API or SDK is buggy or doesn’t provide immediate value, developers gradually remove their implementation causing retention to gradually drop down to 0%.
Unlike the above chart, the below chart shows what bad API retention looks like.
Notice how in this case, the retention goes all the way down to 0%.
Breaking down by segments
Previously, we were looking at our entire user base. However, to dig deeper into why retention drops off, it’s important to find the variants that drive this dip. One way to do this is by grouping your users into buckets. This can be broken down by user properties such as by the country they are from, user acquisition channels, or even product specific information such as the SDK the developer was using.
In this case, we gain more visibility into what’s driving long term retention. While the overall retention curve was relatively flat (i.e. good retention), we see that the majority of our non-United States customers drop down to 0%. Only the United States users continue to leverage the API. We should further investigate why non-US users don’t use the API. Is it due to a high latency? Non-localized documentation? Local laws and regulation which prevent adoption? This might especially be true in financial, healthcare, and other regulated industries.
Defining initial and returning actions
From a business standpoint, the returning action should be something only active users perform and get value out of your product. To properly measure retention you need to define the initial action and a returning action. While not required, initial action usually is the first action a new user makes with your product.
Previously, we were just tracking new users who made their first API Call, and then made any returning API Call. However, we can get much more specific on what those initial or returning actions are.
For example, we may consider a user active only if they returned and made a payment transaction on the API.
Example
Let’s say you’re building a payments API that manages credit card transactions. The main value provided by the API is processing payments via credit card so a merchant gets paid, thus we consider an active user who processes at least one credit card payment via the API per day.
For an API, we may track retention as follows:
First Event Criteria:
- New user who made very first API Call
Returning Event Criteria:
- Came back and made at least one payment transaction
Conclusion
There are many deeper API metrics that you should be tracking if you intend to build a growing API program. However, if you haven’t started taking a data-driven approach yet, these metrics are a good starting point. As you grow your API program, you’ll be tracking other KPIs that can show you the health of your API program. Tools like Moesif API Analytics can help you get started measuring these metrics with just a quick SDK installation.
See Chapter 1 of this series: Mastering API Analytics - the Developer Funnel
Understand Your Platform Retention With Moesif
14 day free trial. No credit card required.
Learn More
Related Articles
API Analytics and Monitoring
Introducing AI Explain
We are thrilled to introduce our newest feature to Moesif - AI Explain - a powerful conversational tool that lets you interact with your API analytics data l...
API Analytics and Monitoring
Monitoring Cost and Consumption of AI APIs and Apps
Discover how Moesif can help manage and optimize your AI-driven APIs with powerful observability tools that track costs and usage efficiently.
API Development
Unlock Deep API Insights with Moesif’s HTTP Body Analytics
Learn how to use Moesif to analyze HTTP bodies.
API Analytics and Monitoring
How Moesif API Observability Compares to Treblle
Learn how Moesif’s API observability platform compares to Treblle.