In Depth Guide to Serverless APIs with AWS Lambda and AWS API Gateway (Part 1)
TL;DR The repository with an example project can be found on GitHub
Building your software products around an API is THE thing for years now and doing it with serverless technology right from the start seems rather intriguing for many reasons — on-demand pricing, auto-scaling, and less operational overhead.
Why
When I read what people are talking about serverless technology, I have the feeling there are still many questions open.
How to
- use a query string or route parameters?
- use the params or body of a POST request?
- set HTTP status codes?
- set a response header?
- call a Lambda from a Lambda?
- store keys for third-party APIs?
So I decided to write an article about how to build an API with serverless technology, specifically AWS Lambda and API-Gateway.
This article is split into two parts.
This one, the first, is about the architecture, setup, and authentication.
The second one is about actual work, uploading images, tagging them with the third-party API, and retrieving the tags and the images later.
What
We will build a simple RESTful image tagging API. It lets us upload images, tags them automatically with a third-party API called Clarifai, and stores the tags and image names into Elasticsearch.
The work-flows I have in mind are:
- Signing up and in
- Uploading an image
- Getting all tags of all images
- Getting all images and their tags
- Getting an image by tag
How
For the API, we use API-Gateway, which is Amazons all-round serverless HTTP solution. It’s optimized for RESTful APIs and works as the entry-point for our system.
Amazon Cognito handles the authentication. Cognito is a managed serverless authentication, authorization, and data synchronization solution. We use it to sign our users up, and in so we don’t have to reinvent the wheel here.
The actual computing work of our API is done by AWS Lambda, a function as a service solution. Lambda is a serverless event-based system that allows triggering functions when something happens, for example, an HTTP request hit our API, or someone uploaded a file directly to S3.
The images are stored in an Amazon S3 bucket. S3 is a serverless object-based storage solution. It allows direct access and uploads of files via HTTP and can, as API-Gateway, be an event source for Lambda.
The tag data and the corresponding image-names are stored in Amazon Elasticsearch Service; an AWS managed version of Elasticsearch. Sadly this is not serverless but built on EC2 instances, but there is a free tier for the first 12 months. Elasticsearch is a very flexible document storage and comes with a powerful query language.
A non-AWS service called Clarifai provides image recognition magic. AWS has its service for this, called Rekognition, but by using Clarifai, we can learn how to store third-party API keys.
The whole infrastructure we build is managed by AWS SAM, the Serverless Application Model. SAM is an extension for AWS CloudFormation that reduces some boilerplate code needed to set up AWS Lambda and API-Gateway resources.
We use AWS Cloud9 as an IDE because it comes with all the tools and permissions pre-installed to use AWS resources.
Prerequisites
- A browser
- An AWS account with a Cloud9 environment. (Setup can be found here in the first 5 steps.)
- Basic JavaScript knowledge
- Basic Lambda, API-Gateway, and SAM knowledge
Setup
Let’s start by getting a basic serverless API going that just implements a login with the help of AWS SAM, API-Gateway and Cognito.
mkdir serverless-api
cd serverless-api
mkdir functions
touch template.yaml
First, we create the folder structure and a template.yaml file that holds the definition of the infrastructure we create with SAM.
Implementing the SAM Template
The content of our SAM template is as follows:
AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: "An example REST API build with serverless technology"
Globals:
Function:
Runtime: nodejs8.10
Handler: index.handler
Timeout: 30
Tags:
Application: Serverless API
Resources:
ServerlessApi:
Type: AWS::Serverless::Api
Properties:
StageName: Prod
Cors: "'*'"
Auth:
DefaultAuthorizer: CognitoAuthorizer
Authorizers:
CognitoAuthorizer:
UserPoolArn: !GetAtt UserPool.Arn
GatewayResponses:
UNAUTHORIZED:
StatusCode: 401
ResponseParameters:
Headers:
Access-Control-Expose-Headers: "'WWW-Authenticate'"
Access-Control-Allow-Origin: "'*'"
WWW-Authenticate: >-
'Bearer realm="admin"'
# ============================== Auth =============================
UserPool:
Type: AWS::Cognito::UserPool
Properties:
UserPoolName: ApiUserPool
LambdaConfig:
PreSignUp: !GetAtt PreSignupFunction.Arn
Policies:
PasswordPolicy:
MinimumLength: 6
UserPoolClient:
Type: AWS::Cognito::UserPoolClient
Properties:
UserPoolId: !Ref UserPool
ClientName: ApiUserPoolClient
GenerateSecret: no
PreSignupFunction:
Type: AWS::Serverless::Function
Properties:
InlineCode: |
exports.handler = async event => {
event.response = { autoConfirmUser: true };
return event;
};
LambdaCognitoUserPoolExecutionPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
FunctionName: !GetAtt PreSignupFunction.Arn
Principal: cognito-idp.amazonaws.com
SourceArn: !Sub "arn:${AWS::Partition}:cognito-idp:${AWS::Region}:${AWS::AccountId}:userpool/${UserPool}"
AuthFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: functions/auth/
Environment:
Variables:
USER_POOL_ID: !Ref UserPool
USER_POOL_CLIENT_ID: !Ref UserPoolClient
Events:
Signup:
Type: Api
Properties:
RestApiId: !Ref ServerlessApi
Path: /signup
Method: POST
Auth:
Authorizer: NONE
Signin:
Type: Api
Properties:
RestApiId: !Ref ServerlessApi
Path: /signin
Method: POST
Auth:
Authorizer: NONE
Outputs:
ApiUrl:
Description: The target URL of the created API
Value: !Sub "https://${ServerlessApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"
Export:
Name: ApiUrl
First, we define a ServerlessApi resource. In a SAM template, this resource is usually created implicitly for us, but when we want to enable CORS or use an authorizer, we need to define it explicitly.
We also set the default authorizer for all routes to be the CognitoAuthorizer and then configure it right away with a Cognito user pool we define later in the template.
Then we define an UNAUTHORIZED gateway response because API-Gateway won’t add CORS headers to our responses on its own. In our Lambda backed routes, we can do this via JavaScript code, but when we dont’t have permissions to call them, this doesn’t help. Gateway responses are a way to add headers directly with API-Gateway.
Next, we define the Cognito related resources:
UserPoolis the part of Cognito that holds our users’ accounts.UserPoolClientis the part of Cognito that allows programmatic interaction with a user pool.PreSignupFunctionis a Lambda function that is called before the actual signup with the user pool happens. It allows us to activate users without the need to send an email to them. It only has a few lines of code, so we inline it.LambdaCognitoUserPoolExecutionPermissiongrant the manged Cognito service the permission to execute ourPreSignupFunction.
Finally, we define a Lambda function that is called by API-Gateway routes. Specifically POST /signup and POST /signin. This AuthFunction also gets the user pool ID and the user pool client ID as environment variables. These values are dynamically generated at deployment time, but environment variables are a way to pass such values from SAM/Cloudformation into a function.
With the Global/Function/Handler we set at the top and the CodeUri in our AuthFunction Properties, we can determine where our JavaScript file has to be and how it has to export a handler function for Lambda.
...
Handler: index.handler
...
CodeUri: functions/auth/
The file has to be called index.js, it has to export a function called handler and it has to be in the functions/auth/ directory.
The Auth property of our endpoint definitions are set to Authorizer: NONE so API-Gateway lets us request the endpoints without the need of a token.
At the end of the file, we have one output called ApiUrl; we use it after the deployment to fetch the actual API URL from CloudFormation.
Implementing the Auth Lambda Function
We created the functions directory at the beginning, so we only need to add an auth directory with an index.js
The index.js holds the following code:
const users = require("./user-management");
exports.handler = async event => {
const body = JSON.parse(event.body);
if (event.path === "/signup") return signUp(body);
return signIn(body);
};
const signUp = async ({ username, password }) => {
try {
await users.signUp(username, password);
return createResponse({ message: "Created" }, 201);
} catch (e) {
console.log(e);
return createResponse({ message: e.message }, 400);
}
};
const signIn = async ({ username, password }) => {
try {
const token = await users.signIn(username, password);
return createResponse({ token }, 201);
} catch (e) {
console.log(e);
return createResponse({ message: e.message }, 400);
}
};
const createResponse = (
data = { message: "OK" },
statusCode = 200
) => ({
statusCode,
body: JSON.stringify(data),
headers: { "Access-Control-Allow-Origin": "*" }
});
It requires a user-management.js to do its work, but we talk about this later. First, let us look at what this file does.
As we said in the template.yaml, It exports a handler function that receives the HTTP request event from API-Gateway when someone sends a POST request to the /signup or /signin endpoints.
Here we can see an answer to one of the most frequent questions.
How to access the request body?
exports.handler = async event => {
const body = JSON.parse(event.body);
...
};
The first parameter of a JavaScript Lambda function holds an event object. When the function is called with an API-Gateway event, this object has a body attribute that holds a string of the request body if the client sent one.
In our case, we expect a JSON with a username and password so we can create new user accounts or sign users into their accounts, so we need to JSON.parse() the body first.
Next, we have two functions, signUp and signIn that use the required user-management module, here called users to do their work. They are both passed username and password as arguments.
createResponse is a utility function that builds a response object. This object is what a Lambda function has to return.
Here, we have an answer to other questions from the beginning.
How to set HTTP status codes?
exports.handler = async event => {
...
return { statusCode: 404 };
};
Every Lambda function needs to return a response object. This object has to have at least a statusCode attribute. Otherwise, API-Gateway considers the request failed.
How to set an HTTP response header?
exports.handler = async event => {
...
return {
statusCode: 200,
headers: {
"Access-Control-Allow-Origin": "*"
}
};
};
The response object our Lambda function has to return can also have a headers attribute, it has to be an object with header names as the objects keys and the header values as the values of the object.
Here, we can see that we need to set CORS headers manually. Otherwise, browsers won’t accept the response.
Now, let’s look at the user-management.js file we required.
global.fetch = require("node-fetch");
const Cognito = require("amazon-cognito-identity-js");
const userPool = new Cognito.CognitoUserPool({
UserPoolId: process.env.USER_POOL_ID,
ClientId: process.env.USER_POOL_CLIENT_ID
});
exports.signUp = (username, password) =>
new Promise((resolve, reject) =>
userPool.signUp(username, password, null, null, (error, result) =>
error ? reject(error) : resolve(result)
)
);
exports.signIn = (username, password) =>
new Promise((resolve, reject) => {
const authenticationDetails = new Cognito.AuthenticationDetails({
Username: username,
Password: password
});
const cognitoUser = new Cognito.CognitoUser({
Username: username,
Pool: userPool
});
cognitoUser.authenticateUser(authenticationDetails, {
onSuccess: result => resolve(result.getIdToken().getJwtToken()),
onFailure: reject
});
});
First, we use the node-fetch package to polyfill the fetch browser API. This polyfill is needed for the amazon-cognito-identity-js package we require next, which does the heavy lifting of talking to the Cognito service for us.
We create a CognitoUserPool object with the help of the environment variables we got from the SAM template and use this object inside the signIn and signUp functions we export.
The most exciting part is in the signIn function that fetches an ID token from the Cognito user pool and returns it.
This token needs to be passed inside an Authorization request header with a Bearer prefix on every request to our API, and it needs to be re-fetched when it expires. The CognitoAuthorizer in the API configuration of our SAM template told API-Gateway how to handle everything else with Cognito.
Now we need to install the packages we used. The node-fetch polyfill package and the amazon-cognito-identity-js package.
For this, we need to go into the functions/auth directory and init an NPM project, then install the packages.
npm init -y
npm i node-fetch amazon-cognito-identity-js
All files inside the functions/auth directory are uploaded to S3 when we deploy the function to Lambda with the rest of our API.
Deploying the API
To deploy our project to AWS, we use the sam CLI tool.
First, we need to package our Lambda function source and upload it to an S3 deployment bucket. We can create this bucket with the aws CLI.
aws s3 mb s3://<DEPLOYMENT_BUCKET_NAME>
The bucket name has to be globally unique, so we need to invent one.
When the creation worked, we need the DEPLOYMENT_BUCKET_NAME to package our Lambda source.
sam package --template-file template.yaml \
--s3-bucket <DEPLOYMENT_BUCKET_NAME> \
--output-template-file packaged.yaml
This command creates a packaged.yaml file that holds URLs to our packaged Lambda sources on S3.
Next, we need to do the actual deployment with CloudFormation.
sam deploy --template-file packaged.yaml \
--stack-name serverless-api \
--capabilities CAPABILITY_IAM
If everything went well, weuse the following command to get the APIs base URL.
aws cloudformation describe-stacks \
--stack-name serverless-api \
--query 'Stacks[0].Outputs[?OutputKey==`ApiUrl`].OutputValue' \
--output text
The URL should look something like this:
https://<API_ID>.execute-api.<REGION>.amazonaws.com/Prod/
This URL can now be used to issue POST requests to the /signup and /signin endpoints we created.
Conclusion
This article was the first of two articles about creating serverless APIs on AWS. We talked about the motivations to do so, the AWS services we need to get things done and implemented token-based authentication with the help of AWS Cognito.
We also answered a few of the most pressing questions that arise when building an API with API-Gateway and Lambda.
In the next part, we implement two Lambda functions that allow us to work with the API.
ImagesFunctionis responsible for creating upload links for an S3 bucket.TagsFunctionhandles the S3 uploads, third-party API integration, and the listing of created tags.
Related Articles
API Analytics and Monitoring
Introducing AI Explain
We are thrilled to introduce our newest feature to Moesif - AI Explain - a powerful conversational tool that lets you interact with your API analytics data l...
API Analytics and Monitoring
Monitoring Cost and Consumption of AI APIs and Apps
Discover how Moesif can help manage and optimize your AI-driven APIs with powerful observability tools that track costs and usage efficiently.
API Development
Unlock Deep API Insights with Moesif’s HTTP Body Analytics
Learn how to use Moesif to analyze HTTP bodies.
API Analytics and Monitoring
How Moesif API Observability Compares to Treblle
Learn how Moesif’s API observability platform compares to Treblle.