How to Monitor Azure API Management Performance with the Moesif Plugin
Azure API Management (APIM) is a powerful platform that enables you to publish and scale APIs while ensuring they are secured. One of the great features of Azure APIM is that you can add plugins and transforms to your APIs within an API Management service instance without any code change or restarts.
These capabilities are deployed using XML Policies which are a collection of statements. Moesif API Observability can be added in just a few minutes using an XML policy for APIM which makes it easy get visibility into API calls, even ones that are rejected and never reach your underlying service.
What is Azure API Management?
Azure API Management is a fully managed service that enables developers to expose their APIs to both external and internal customers. It provides a comprehensive set of tools and services for creating, publishing, and managing APIs, ensuring they are secure, scalable, and easy to use. As a key component of Azure API, it allows developers to build and manage APIs efficiently, leveraging the power of cloud-based services for robust and secure API management.
Azure API Management Components
Azure API Management consists of several integral components that work together to provide a comprehensive API management solution:
-
API Gateway: This acts as a facade to the backend services, allowing API providers to abstract API implementations and evolve backend architecture without impacting API consumers. It handles all incoming API calls, applies policies, and routes them to the appropriate backend services.
-
Management Plane: This component provides full access to the API Management service capabilities. It allows API providers to create and manage APIs, configure API policies and security settings, and monitor and analyze API usage and performance.
-
Developer Portal: An automatically generated, fully customizable website that documents your APIs. It enables API consumers to discover, subscribe to, and learn how to consume APIs in their applications, fostering a community of developers around your APIs.
Integration with Azure Services
Azure API Management integrates seamlessly with many complementary Azure services to create robust enterprise solutions. Key integrations include:
-
Azure Active Directory: For robust authentication and authorization mechanisms.
-
Azure Storage: For efficient storage and management of API data.
-
Azure Cosmos DB: For handling large-scale API data with high availability and low latency.
-
Azure Functions: For building serverless API backends, enabling scalable and event-driven API solutions.
-
Azure Logic Apps: For integrating APIs with other Azure services and external systems, facilitating complex workflows and automation.
These integrations empower developers to build comprehensive API solutions that leverage the full potential of Azure’s cloud services, ensuring scalability, security, and efficiency.
API Management Tiers and Pricing
Azure API Management is offered in a variety of pricing tiers to cater to different customer needs. Each tier provides a distinct combination of features, performance, capacity limits, scalability, SLA, and pricing, suitable for various scenarios:
-
Developer Tier: Ideal for development and testing environments, this tier offers limited features and capacity, making it cost-effective for non-production use.
-
Basic Tier: Suitable for small-scale production environments, it provides standard features and capacity, balancing cost and functionality.
-
Standard Tier: Designed for medium-scale production environments, this tier offers advanced features and capacity, supporting more extensive API management needs.
-
Premium Tier: Perfect for large-scale production environments, it delivers high-performance features and capacity, ensuring robust and scalable API management.
What is Moesif API Observability for Azure APIM?
API Observability enables engineering and business to deeply understand how their APIs are used and identify what needs to be fixed before customers email support and overwhelm your team. API management supports a variety of functionalities as part of a platform-as-a-service model, ensuring compliance and security. Unlike classic API monitoring which usually probes an endpoint for typical “Red, Yellow, Green” status, API Observability enables leaders to observe any behavior happening with the API. Some examples include:
- Product owners to understand API usage and what to focus on
- Engineering leaders to stay informed of and troubleshoot API issues
- Security researchers to investigate API threats and prevent them
How does this solution works
In order for API Observability to work, a monitoring agent needs to passively log API traffic to an observability service. This can be a custom build on a data warehouse like Azure Synapse Analytics or or a turnkey solution like Moesif. An API Gateway like Azure APIM provides a natural point to centralize API logs. Otherwise, each service would need to have it’s own SDK.
This solution is deployed using an Azure Resource Manager Template, which automatically adds a few components to your Azure subscription:
| Component | Purpose |
|---|---|
| API Management Logger | Captures API logs from your API Management instance and inserts them into the EventHub |
| Azure EventHub | Buffers the raw API logs until ready to be consumed |
| Azure WebApp | Runs the ApimEventProcessor app which reads from EventHub and sends logs to Moesif in batches |
A diagram is below showing how the solution is deployed in your Azure subscription.
Once the API logs are received by Moesif, the rest of the data processing and analytics is handled by the service.
Use cases
Understand Customer API usage
A goal for API analytics is to gain an understanding on who is using your APIs and how they use them. To enable this, the integration will automatically associate API calls to specific user profiles. By default, The XML policy will also extract user info like the User Id, First Name, and Email from the context.User object and saved as part of a user profile in Moesif. You can always add additional user properties using Moesif’s user tracking API.
A critical report is understanding which customers are using your APIs the most. Because we are tracking name and email, we can open a report in Moesif showing weekly API traffic by company name.
Troubleshoot issues
With high-cardinality, high-dimension API observability, you can slice and dice your API logs by any number of fields such as the URI Route, HTTP headers, and even fields in the payload enabling you to drill into issues impacting customers. One such metric we recommend monitoring is the 90th percentile. Unlike average latency, by looking at 90th percentile latency, you can better see large variations in your latency which is usually worse for customers than an API that’s consistently high latency. An API with seamlessly random latency spikes can wreak havoc in their own services.
To do this, go to Events -> Time series and then select the metric P90 Latency. You can also understand this broken down by route or service. To do so, add a group by “Request URI.” Moesif consolidate routes such that /items/1 and /items/2 will show up as /items/:id in the UI which makes it easier for your analysis.
Research threats
As you expose more APIs to the internet used by customers, partners, and single page apps, your security risk goes up. Traditional mechanisms like browser fingerprinting and captchas don’t work so you need to leverage advanced user behavior analytics to find suspicious users.
A common API security threat is not having proper protection from data scraping and intentional API abuse. An API provides direct access to your data which a hacker can use to scrape. One way to detect customers abusing your API is to look at the amount of data accessed per user. To create this metric, add a summation of response.headers.Content-Length and then group it by user’s name:
How to set up Azure APIM with Moesif API Observability
1. Start Azure Resource Deployment
As an API provider, click the below button to start a Custom deployment with the Moesif Azure Resource Template.
2. Configure Parameters
Within the Azure Template Deployment panel, set the following properties:
-
Set Resource group to the same resource group that contains your exiting Azure APIM instance. This ensures the APIM logger,
moesif-log-to-event-hub, is automatically created for you. -
Set Moesif Application Id to the one displayed after logging into your Moesif account. You can create a free one on Moesif’s website
-
Set Existing Api Mgmt Name to the name of your Azure APIM instance. If blank, you will need to manually create the APIM logger.
Once done, click the Review+create button at the bottom and finish the template creation wizard.
Occasionally, Azure reports a failed deployment due to slow propagation of new DNS settings even though everything was deployed successfully. We recommend proceeding with rest of process. If you still have issues after last step, view troubleshooting.
3. Add XML Policy
Within the Azure portal, navigate to your existing Azure API Management instance. Then, add the below XML policies to all products or APIs that you want API logging enabled.
It’s recommended to add the XML policy globally for all APIs. Then, use Moesif dynamic sampling if you want to create rules that selectively sample or suppress data collection. Rules are dynamically enabled based on specific customer behaviors, regex rules, and more.
More info on editing APIM policies is available on the Azure docs
<policies>
<inbound>
<base />
<set-variable name="moesif-message-id" value="@(Guid.NewGuid())" />
<log-to-eventhub logger-id="moesif-log-to-event-hub" partition-id="0">@{
var body = context.Request.Body?.As<string>(true);
var MAX_BODY_EH = 145000;
var origBodyLen = (null != body) ? body.Length : 0;
if (MAX_BODY_EH < origBodyLen){ body = body.Remove(MAX_BODY_EH); }
var headers = context.Request.Headers
.Where(h => h.Key != "Ocp-Apim-Subscription-Key")
.Select(h => string.Format("{0}: {1}", h.Key, String.Join(", ", h.Value).Replace("\"", "\\\""))).ToArray<string>();
var jwtToken = context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt();
var userId = (context.User != null && context.User.Id != null) ? context.User.Id : (jwtToken != null && jwtToken.Subject != null ? jwtToken.Subject : string.Empty);
var cru = new JObject();
if (context.User != null) {
cru.Add("Email", context.User.Email);
cru.Add("Id", context.User.Id);
cru.Add("FirstName", context.User.FirstName);
cru.Add("LastName", context.User.LastName);}
var crus = System.Convert.ToBase64String(Encoding.UTF8.GetBytes(cru.ToString()));
var requestBody = (body != null ? System.Convert.ToBase64String(Encoding.UTF8.GetBytes(body)) : string.Empty);
return new JObject(
new JProperty("event_type", "request"),
new JProperty("message-id", context.Variables["moesif-message-id"]),
new JProperty("method", context.Request.Method),
new JProperty("ip_address", context.Request.IpAddress),
new JProperty("uri", context.Request.OriginalUrl.ToString()),
new JProperty("user_id", userId),
new JProperty("contextRequestUser", crus),
new JProperty("company_id", ""),
new JProperty("request_headers", string.Join(";;", headers)),
new JProperty("request_body", requestBody),
new JProperty("contextTimestamp", context.Timestamp.ToString("o")),
new JProperty("metadata", $@"")
).ToString();}</log-to-eventhub>
<set-variable name="sent-moesif-request" value="@(true)" />
</inbound>
<backend>
<forward-request follow-redirects="true" />
</backend>
<outbound>
<base />
<choose>
<when condition="@(context.Variables.ContainsKey("sent-moesif-request") && !context.Variables.ContainsKey("sent-moesif-response"))">
<log-to-eventhub logger-id="moesif-log-to-event-hub" partition-id="1">@{
var body = context.Response.Body?.As<string>(true);
var MAX_BODY_EH = 145000;
var origBodyLen = (null != body) ? body.Length : 0;
if (MAX_BODY_EH < origBodyLen){ body = body.Remove(MAX_BODY_EH);}
var headers = context.Response.Headers.Select(h => string.Format("{0}: {1}", h.Key, String.Join(", ", h.Value).Replace("\"", "\\\""))).ToArray<string>();
var responseBody = (body != null ? System.Convert.ToBase64String(Encoding.UTF8.GetBytes(body)) : string.Empty);
return new JObject(
new JProperty("event_type", "response"),
new JProperty("orig_body_len", origBodyLen),
new JProperty("message-id", context.Variables["moesif-message-id"]),
new JProperty("status_code", context.Response.StatusCode),
new JProperty("response_headers", string.Join(";;", headers)),
new JProperty("contextTimestamp", context.Timestamp.Add(context.Elapsed).ToString("o")),
new JProperty("response_body", responseBody)
).ToString();}</log-to-eventhub>
<set-variable name="sent-moesif-response" value="@(true)" />
</when>
</choose>
</outbound>
<on-error>
<base />
<choose>
<when condition="@(context.Variables.ContainsKey("sent-moesif-request") && !context.Variables.ContainsKey("sent-moesif-response"))">
<log-to-eventhub logger-id="moesif-log-to-event-hub" partition-id="1">@{
var body = context.Response.Body?.As<string>(true);
var MAX_BODY_EH = 145000;
var origBodyLen = (null != body) ? body.Length : 0;
if (MAX_BODY_EH < origBodyLen){ body = body.Remove(MAX_BODY_EH);}
var headers = context.Response.Headers.Select(h => string.Format("{0}: {1}", h.Key, String.Join(", ", h.Value).Replace("\"", "\\\""))).ToArray<string>();
var responseBody = (body != null ? System.Convert.ToBase64String(Encoding.UTF8.GetBytes(body)) : string.Empty);
return new JObject(
new JProperty("event_type", "response"),
new JProperty("orig_body_len", origBodyLen),
new JProperty("message-id", context.Variables["moesif-message-id"]),
new JProperty("status_code", context.Response.StatusCode),
new JProperty("response_headers", string.Join(";;", headers)),
new JProperty("contextTimestamp", context.Timestamp.Add(context.Elapsed).ToString("o")),
new JProperty("response_body", responseBody)
).ToString();}</log-to-eventhub>
<set-variable name="sent-moesif-response" value="@(true)" />
</when>
</choose>
</on-error>
</policies>
4. Success!
With the Azure APIM integration done, you should see your API logs show up in Moesif. Make a few calls against your API Gateway domain and see them show up in Moesif’s event log in real-time. You should see the status code, URL, and other HTTP parameters captured like the below screenshot:
Identifying users
API calls are associated to users using the field user_id. The default XML policy extracts this from the context.User.Id or the JWT Token using the following logic:
var jwtToken = context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt();
var userId = (context.User != null && context.User.Id != null) ? context.User.Id : (jwtToken != null && jwtToken.Subject != null ? jwtToken.Subject : null);
You can modify what the userId is by changing these lines of code.
Adding user metadata
By default, the XML policy also saves some helpful user properties like Email and FirstName using the following code in the XML Policy:
if (context.User != null) {
cru.Add("Email", context.User.Email);
cru.Add("FirstName", context.User.FirstName);
cru.Add("LastName", context.User.LastName);
}
var crus = System.Convert.ToBase64String(Encoding.UTF8.GetBytes(cru.ToString()));
These will be saved with the user profile in Moesif that matches the defined userId. You can add additional fields from the context.User to meet your requirements by changing these lines of code.
Adding Event metadata
You can also save event metadata. Unlike user metadata, event metadata is specific to each API transaction and can contain helpful info not already logged by Moesif such as trace ids or environment variables. The metadata field should be a JSON encoded string.
Sampling requests
This integration also supports dynamic sampling. This enables you to selectively sample API calls based on user behavior to save on your Moesif subscription cost. Moesif will still extrapolate the original metrics.
Advanced User Behavior API Analytics
You can leverage your integration beyond just looking at API calls in isolation and stitch your entire customer journey together. This approach makes it easier to see things like funnel reports on “Time to First Hello World” and “Time to Value.”
Track user actions in your UI such as “Signed In” or “Viewed Docs” and start tracking user actions in your UI like “Signed In” or “Viewed Docs”. This makes it easier to slice and dice API usage by customer traffics. In order to do so, add the moesif-browser-js to your UI and call the track method:
moesif.track('Clicked Sign Up', {
button_label: 'Get Started',
sign_up_method: 'Google SSO'
});
Once done, the first thing you should do is generate a funnel report. In the below report, we created a funnel analysis composing of three steps.
- The first step is a customer signing into your web application (a user action).
- The second step is a single payment transaction via the API. Thus moving from step 1 to step 2 shows the conversion rate of sign ups to the first API call.
- The third step is over 100 payment transactions. For this example, we consider this the “aha” moment demonstrating customer value. Moving from step 2 to step 3 shows the drop off of customers who made API calls who actually got to see real value.
Conclusion
API observability is critical for engineering and business leaders to make informed decisions on what to focus on and where issues are. While you can roll your own API gateway, data processing pipeline, and a data warehouse, this can create a massive time sink for your engineering team. Using fully managed services like Azure API Management API Gateway and Moesif API Analytics can help you scale without being held back by legacy infrastructure.
Related Articles
API Development
WSO2 APK + Moesif API Analytics: Drive API Performance and Adoption
Learn how you can drive API performance and adoption in WSO2 APK with Moesif API analytics plugin.
API Development
WSO2 Choreo + Moesif API Analytics: Optimize API Performance And Drive Adoption
Learn how you can optimize API performance and drive adoption in WSO2 Choreo with Moesif API analytics plugin.
Podcasts
APIs Over IPAs 15: Customer Observability with Mike Amundsen, Amundsen.com
Derric Gilling sits down with Mike Amundsen, a leader in the API ecosystem, to explore the role of observability in API design and customer experience.
API Development
Achieving API observability in Kong Konnect with Moesif API analytics plugin
Learn how you can achieve API observability and monitoring in Kong Konnect with Moesif API analytics plugin.
