Integrating with AWS API Gateway

AWS API Gateway is an extremely popular way to quickly and securely manage your RESTful APIs. Moesif is a great addition on top of AWS API Gateway to cover everything from analytics to helping you set up metered billing.

Building your API is just the first step. Once built, you need to keep customers and services using the API happy. Moesif is the tool to help you do exactly that. Integrating Moesif with your AWS API Gateway instance can be done in a few simple steps.

How It Works

The integration works by Logging API calls to a Kinesis Data Firehose which sends access logs in batches to Moesif. Besides tracking typical API parameters like URI route and status code, the integration also tracks the user identity which makes them perfect for user behavior analytics tools like Moesif.

Diagram of Moesif AWS API Gateway logging architecture

Because you are monitoring at the API gateway itself, you also gain visibility into rejected requests that never reach your underlying service such as AWS Lambda or EC2.

Step 1: Create Your Moesif Account

If you don’t already have a Moesif account, You’ll want to sign up. Once you’re registered you’ll need to add in a few details.

Moesif create account first screen

These include:

  • Organization Name
  • the name of your organization or company
  • App Name
  • the name of the app you are integrating with Moesif
  • What best matches your role?
  • Here you can describe what your current role is at your organization. This helps us to assist you with relevant hints and tips in the future.
  • What are you hoping to achieve?
  • Here you can let us know how you'll use Moesif to help your organization or project.

Once you’ve filled out these details, click Next. After this, you’ll be presented with a brief overview of Moesif.

Moesif create account overview screen

Click Next once again.

Step 2: Create the Kinesis Firehose for Moesif

After the Overview screen, you’ll be presented with the Quick Install screen. Here you can pick from any different server integrations and plugins. For AWS API Gateway we want to make sure AWS is selected in the left pane and that the AWS API Gateway walkthrough is displayed on the right.

Moesif config screen with AWS Gateway integration info displayed

Once selected, follow Step 1 on the screen and create the Kinesis Firehose for Moesif. To do this, click on the Launch Stack button in the instructions.

Launch Stack button

You’ll then be redirected to the CloudFormation screen to create the stack in AWS. The details will be pre-filled on the CloudFormation screen. The only thing you’ll need to do is check off the box allowing for IAM Resources to be created. The last step is clicking Create stack.

CloudFormation create stack screen

Your stack will then be created along with your Kinesis Firehose.

CloudFormation stack created

Next, we will grab our Kinesis Firehose ARN. To do that, in the search at the top of the screen, type in “Kinesis” nd click on the Kinesis entry.

Kinesis search

Once on the Kinesis home screen, you’ll click on the Data Firehose entry.

Kinesis data firehose

The next screen will show you the available Delivery Steams. Click on the one that you just created with Cloudformation (the name will be amazon-apigateway-api-logs-to-moesif).

Kinesis data firehose for Moesif

Lastly, we will grab the ARN so that we can plug it into the gateway configuration. Copy the ARN to your clipboard for later.

Data Firehose ARN

Step 3: Add the Data Firehose to Your Gateway Configuration

Now that your Kinesis Data Firehose is created, you can add it to your gateway so API analytics are forwarded to Moesif. Navigate to your AWS API Gateway instance. Once there, click on Stages in the left-side menu. On the Stages screen, click on the Stage you’d like to add the Kinesis Firehose to. Then, you’ll click on the Logs/Tracing tab. On this screen, check off Enable Access Logging and in the appearing fields add your Firehose ARN from earlier. The Log Format field can be filled in with the following JSON:

{   "apiId": "$context.apiId",
    "requestId": "$context.requestId",
    "requestTime": "$context.requestTime",
    "protocol": "$context.protocol",
    "httpMethod": "$context.httpMethod",
    "resourcePath": "$context.resourcePath",
    "requestHostHeader": "$context.domainName",
    "requestUserAgentHeader": "$context.identity.userAgent",
    "ip": "$context.identity.sourceIp",
    "status": "$context.status",
    "responseLength":"$context.responseLength",
    "durationMs": "$context.responseLatency",
    "caller": "$context.identity.caller",
    "user": "$context.identity.user",
    "principalId": "$context.authorizer.principalId",
    "cognitoIdentityId": "$context.identity.cognitoIdentityId",
    "userArn": "$context.identity.userArn",
    "apiKey": "$context.identity.apiKey"
}

The settings should now look like this:

Enable access logging on gateway

Once these settings are in place, you’ll click Save at the bottom of the screen.

Save Access Logging settings

Step 4: Send a Request to Your Endpoint

Using your application or favorite tool, send a request to your endpoint. Once sent, give it a few minutes to propagate into Moesif.

Step 5: Confirm Your Integration

After the request is sent, Moesif should receive the event and data. This may take a few moments for the first request to reflect in the system once sent. Once received, Moesif will confirm that you are ready to move forward. The previous prompt will turn green and show “Data Received!”.

Moesif data received onboarding screen

Lastly, scroll to the bottom of the screen and click Next.

Moesif next button

Step 6: Invite Your Team

The final step in the setup is to invite your team. The next screen is where you can do this quickly and easily.

Moesif invite team screen

If you’d like to add a team member, simply add their email and click Send Invite.If you don’t have any team members yet or want to skip this step, you can always add team members later.

Once you’ve added all the team members you want, click Finish.

Explore Moesif

You’ve now successfully integrated your API application with Moesif. You should now see the Moesif dashboard in the browser, ready to explore.

Moesif dashboard with event received

Now that your API is integrated and moving data into Moesif you can start to explore. Check out our docs to find everything from API analytics, monitoring and alerts, and more!

Optional: Sample API Events

By default, the AWS API Gateway integration sends every access log record to Moesif. Moesif provides an AWS Lambda function that samples events. You add it to your Firehose delivery stream as a data transformation. Firehose then runs the function on each batch of records before it delivers the batch to Moesif.

The function keeps or drops each event based on sampling rules that you manage in Moesif. Your API Gateway access logging settings and the delivery stream’s destination stay the same.

Sampling rules are part of Dynamic Sampling. Dynamic Sampling is available in Moesif Enterprise plans.

Why Sample Events

Sampling reduces the number of events that reach Moesif. Use it to do the following:

  • Drop traffic with no analytical value, such as health checks.
  • Keep a representative fraction of high-volume traffic.
  • Reduce the number of events that count against your event quota.

How Moesif Maintains Accuracy in Counts

When the function keeps an event at a sample rate below 100%, it adds a weight field to the event. The weight is the number of original events that the kept event represents. Moesif uses the weight to calculate the true volume. Your counts, volume charts, and trends stay accurate even though Moesif stores fewer events.

The following table shows how the weight works for 1,000 events at different sample rates:

Events Sent Sample Rate Events Stored in Moesif Weight of Each Event Count That Moesif Reports
1,000 100% 1,000 1 1,000
1,000 50% About 500 2 About 1,000
1,000 10% About 100 10 About 1,000

Use a sample rate that divides 100 evenly, such as 50, 25, 20, 10, 5, 4, 2, or 1. The function rounds the weight down. Other rates report slightly fewer events than you sent. For example, a rate of 30% gives a weight of 3, so Moesif reports about 90% of the true count.

Sampling has the following trade-offs:

  • Fewer individual requests: Moesif never receives a dropped event. You can’t look up details about dropped requests later.
  • Per-event decisions: The function samples each event independently, not per user or session. It doesn’t keep or drop a user’s requests as a group. A trace can have gaps as a result.

Deploy the Lambda Function

The function uses only the Python standard library. You don’t need to package any other dependencies.

To deploy the function, follow these steps:

  1. Download lambda_function.py from the GitHub repository for Moesif’s Lambda function for Firehose data transformation.
  2. Package the function:

    zip function.zip lambda_function.py
    
  3. Create a Lambda function from function.zip with the following settings:
    • Runtime: Python 3.12 or later
    • Handler: lambda_function.lambda_handler
    • Memory: 512 MB
    • Timeout: at least 30 seconds
  4. Set the function’s environment variables. For more information, see Set the Environment Variables.

The function samples a full 6 MB buffer in well under a second. The timeout needs to cover a fetch of your sampling rules from Moesif that you specify with CONFIG_FETCH_TIMEOUT_SECONDS. If your buffers are large or you want a larger safety margin, increase the timeout. Firehose supports a function timeout of up to 5 minutes.

Set the Environment Variables

The function reads all of its settings from environment variables. Only MOESIF_APPLICATION_ID is required.

The following table describes each environment variable:

Variable Required Default Description
MOESIF_APPLICATION_ID Yes None Your Moesif Application ID. The function uses it to fetch your sampling rules. If you don’t set it, the function keeps every event.
MOESIF_BASE_URI No https://api.moesif.net The base URL of the Moesif API. Override it for another region or a proxy.
CONFIG_REFRESH_SECONDS No 60 How often, in seconds, a running function checks Moesif for updated rules.
CONFIG_FETCH_TIMEOUT_SECONDS No 6 The total time, in seconds, that a rules fetch can take, including retries.
DEBUG No Off Set to true for verbose logging.

To find your Application ID, see the installation steps in the AWS API Gateway integration documentation.

Each running instance of the function refreshes its rules independently. If Firehose runs many instances at the same time, increase CONFIG_REFRESH_SECONDS to reduce the number of requests to Moesif.

Attach the Lambda Function to Your Delivery Stream

To attach the function to your Firehose delivery stream, follow these steps:

  1. In the AWS Console, open Amazon Data Firehose.
  2. Select your delivery stream, such as amazon-apigateway-api-logs-to-moesif.
  3. In the Transform source records with AWS Lambda section, select the Enable data transformation checkbox.
  4. Choose the Lambda function that you deployed.
  5. Keep the default buffer size of 1 MB and buffer interval of 60 seconds. These values are a good starting point.
  6. Save your changes.
  7. Update the delivery stream’s AWS Identity and Access Management (IAM) role to grant the lambda:InvokeFunction permission on the function. The role that the Moesif CloudFormation stack creates doesn’t include this permission.

The delivery stream stays active while you update it. The changes usually take effect within a few minutes. After that, Firehose sends each batch of records through the function before delivering the batch to Moesif. For more information about the transformation settings, see Transform source data in Amazon Data Firehose.

Configure Sampling Rules in Moesif

To create or change sampling rules, use the Dynamic Sampling page in Moesif. For instructions, see Dynamic Sampling.

The function fetches your rules at runtime. A running function applies rule changes within CONFIG_REFRESH_SECONDS. You don’t need to redeploy or edit the function.

Sampling rules apply the same way as they do for an SDK integration. The function resolves a sample rate for each event in the following order:

  1. Regex rules, in order. The first rule whose conditions all match sets the rate.
  2. The user’s sample rate.
  3. The company’s sample rate.
  4. The global sample rate.

Place regex rules with a sample rate of 100% or 0% before regex rules with any other rate.

API Gateway access log records use different field names from Moesif events. The function reads each rule field from the matching access log field. The following table lists the mapping:

Moesif Field Access Log Field
request.verb httpMethod
request.route resourcePath
request.uri resourcePath
request.ip_address ip
response.status status
User ID The first value present in user, apiKeyId, principalId, cognitoIdentityId, or caller
Company ID companyId

A regex rule on any other field doesn’t match API Gateway access log records.

Monitor Sampling

Each invocation of the function writes one summary line to Amazon CloudWatch Logs, such as the following:

{
  "msg": "firehose_sampling_summary",
  "records_in": 500,
  "events_in": 500,
  "events_kept": 61,
  "events_dropped": 439,
  "records_dropped": 439,
  "config_source": "moesif:/v1/config",
  "config_valid": true
}

To see hourly totals of sent and dropped events, run the following query in CloudWatch Logs Insights:

fields @timestamp, events_in, events_kept, events_dropped
| filter msg = "firehose_sampling_summary"
| stats sum(events_in) as sent, sum(events_dropped) as dropped by bin(1h)

Watch for the following values in the summary line:

  • config_valid: false: No rules are in effect, so the function keeps every event. Either MOESIF_APPLICATION_ID isn’t set or the function couldn’t reach Moesif when a new instance started. Consider an alarm for this value.
  • events_dropped: 0 when you expect dropped events: A rule usually references a field that your stream doesn’t carry. For the supported fields, see Configure Sampling Rules in Moesif.

Before you rely on sampled counts for reporting or billing, confirm in Moesif that the calculated volume matches what you expect.

Failure Behavior

The function doesn’t stop delivery when something goes wrong:

  • If the function can’t reach Moesif, it keeps the existing rules. The function tries again at the next refresh interval.
  • If the function has no rules, it keeps every event.
  • If the function can’t process a record, it passes the record through unchanged. It never marks a record as ProcessingFailed, so no records go to the delivery stream’s error output.

The function samples and filters events only. It doesn’t enforce governance rules. For governance rules with this integration, see Supporting Quotas and Governance Rules.

From Moesif

Other

Updated: