Build a highly scalable REST API in 100 lines
Posted on Wed 02 December 2020 in programming
Today I'll be walking through how to build a REST API endpoint with the following features:
- Automatic TLS/SSL encryption
- Accessible by both front and back-end applications
- Horizontally scalable
- API key validation and limiting
- Persistent storage
- Automated deployments
All in about 100 lines of code. It can be a bit more if you wish to include verbose comments, or it can be a bit less if you discount the starting boilerplate code. The line count itself is not important, but serves as a proxy to illustrate the little amount of effort it takes to start a modern REST API if you choose the right tools to do so.
The tools?
AWS Lambda, the Serverless Framework, and node.js.
Lambda bills itself as a pay-for-usage way of running code without setting up a server to run it. Even so, you'll still need to configure the environment through what's known as a CloudFormation template - a configuration file written in either JSON or YAML. CloudFormation templates will get verbose and tedius to write for anything beyond a sample application. That's where the next framework comes in.
Serverless is a provisioning framework meant to make deploying lambdas a more
pleasant experience. It works with a number of cloud providers, defaulting to
AWS. You will still end up needing to write parts of Cloudformation here and
there (read: make sure to have the docs handy), but the serverless.yaml
configuration file is miles easier to understand and work with than vanilla
CloudFormation.
node.js, but this can be interchangeable as both Serverless and AWS allow lambdas to be written in a great many languages.
The API?
We will build an API with one endpoint, /hello
. It will require an API key to
access, and when invoked correctly will return a JSON payload greeting the
person (customizeable through the name
url parameter). The endpoint will also
keep track of the number of people who have called the endpoint with a given
name, and will tell them how many other people of the same name have come
before.
The code?
I'm glad you're excited, but rather than showing you all the code up front, I think it will be helpful to walk through the changes step by step as if reading a series of git commits.
7731ca0
Initial commit
The boilerplate code we're left with after going through the serverless getting started guide. Some unnecessary lines have been removed for the sake of brevity.
handler.js
module.exports.hello = async event => {
return {
statusCode: 200,
body: JSON.stringify(
{
message: 'Go Serverless v1.0! Your function executed successfully!',
input: event,
},
null,
2
),
};
};
A single exported async function; the entrypoint for our lambda. Maybe it's
obvious to say this, but every time the lambda is invoked, this function is
called. The function itself accepts one parameter, event
, and returns a JSON response with a
success message.
Up to 3 parameters can be passed to the function, depending on signature, but in this case we only need the first. AWS has more documentation on the node.js handler for the curious.
serverless.yml
# For full config options, check the docs:
# docs.serverless.com
service: SERVICE_NAME_HERE
# app and org for use with dashboard.serverless.com
app: APP_NAME_HERE
org: SERVERLESS_ACCOUNT_HERE
# You can pin your service to only deploy with a specific Serverless version
# Check out our docs for more details
frameworkVersion: '2'
provider:
name: aws
runtime: nodejs12.x
functions:
hello:
handler: handler.hello
Most important right now is the functions
section, which defines a hello
lambda whose entrypoint is the hello
function of handler.js
. At this point,
we can already deploy the code to AWS through serverless deploy
and invoke
it with serverless invoke -f hello
When deploying, serverless converts the above yaml file to a CloudFormation template and executes that. Neat, huh? If you're curious and want to take a peek behind the curtains, run
serverless package
and check out the CloudFormation files that are generated in the.serverless
folder.
8ee5621
Make endpoint public
--- a/blocks/lambda-webhook-callback/serverless.yml
+++ b/blocks/lambda-webhook-callback/serverless.yml
@@ -63,10 +63,10 @@ functions:
functions:
hello:
handler: handler.hello
+ events:
+ - http:
+ path: hello
+ method: get
These four lines allow us to call our lambda externally through an https endpoint.
$ curl 'https://sdlfkj3408jjlk.execute-api.us-east-1.amazonaws.com/dev/hello'
{
"message": "Go Serverless v1.0! Your function executed successfully!",
"input": {
...
}
}
Serverless achieves this by automatically configuring an API Gateway entry for the application.
d169028
Greet based on url param
--- a/blocks/lambda-webhook-callback/handler.js
+++ b/blocks/lambda-webhook-callback/handler.js
@@ -1,12 +1,16 @@
'use strict';
module.exports.hello = async event => {
+ let name = 'anonymous'
+ if (event.queryStringParameters && event.queryStringParameters.name) {
+ name = event.queryStringParameters.name
+ }
+
return {
statusCode: 200,
body: JSON.stringify(
{
- message: 'Go Serverless v1.0! Your function executed successfully!',
- input: event,
+ message: `Hello ${name}!`
},
null,
2
Now that our lambda can be called by the API Gateway, we get all sorts of
interesting properties through the event
parameter. In this case, we want to
know if the user passed in a name
url parameter.
$ curl 'https://sdflkj398987.execute-api.us-east-1.amazonaws.com/dev/hello'
{
"message": "Hello anonymous!"
}
$ curl 'https://sdflkj398987.execute-api.us-east-1.amazonaws.com/dev/hello?name=brooks'
{
"message": "Hello brooks!"
}
API Gateway docs with example event
payload
f486d35
Require api keys
--- a/blocks/lambda-webhook-callback/serverless.yml
+++ b/blocks/lambda-webhook-callback/serverless.yml
@@ -23,6 +23,16 @@ frameworkVersion: '2'
provider:
name: aws
runtime: nodejs12.x
+ apiKeys:
+ - testkey1
+ - testkey2
@@ -67,6 +77,7 @@ functions:
- http:
path: hello
method: get
+ private: true
API keys are configured for the whole project, so if you end up having more
than one function/lambda they will all share the same set of API keys.
Setting our function to private will now only allow traffic through if they
have an X-Api-Key
header with a valid API key. Serverless has great
documentation on setting up
keys
as well, there's many more ways to customize them than I've shown here.
$ curl 'https://sdflkj398987.execute-api.us-east-1.amazonaws.com/dev/hello'
{"message":"Forbidden"}
$ curl 'https://sdflkj398987.execute-api.us-east-1.amazonaws.com/dev/hello' \
-H 'x-api-key: sdlfkjo4582730sdflskdfj034lskdfj'
{
"message": "Hello anonymous!"
}
b2d9941
Set CORS for endpoint
--- a/blocks/lambda-webhook-callback/serverless.yml
+++ b/blocks/lambda-webhook-callback/serverless.yml
@@ -78,6 +78,7 @@ functions:
path: hello
method: get
private: true
+ cors: true
This is a small code change, but the impact is huge. With CORS enabled, our API can now be queried from any front end application.
What is CORS? Cross-Origin-Resource-Sharing. In brief, it's a security check
that the browser makes when making "special" requests to endpoints on a
different domain than itself. Our API requires the special x-api-key
header,
which the browser identifies as "special," so it fires off an OPTIONS
http
request against the endpoint to check if this action is allowed or not. If
you're not familiar with what CORS is or how it works, the Mozilla web docs
have a comprehensive
article to
understanding CORS.
22d2412
Set usage plans for keys
--- a/blocks/lambda-webhook-callback/serverless.yml
+++ b/blocks/lambda-webhook-callback/serverless.yml
@@ -26,13 +26,26 @@ provider:
apiKeys:
- - testkey1
- - testkey2
+ - free:
+ - testkey1
+ - paid:
+ - testkey2
+ usagePlan:
+ - free:
+ quota:
+ limit: 5
+ period: DAY
+ throttle:
+ burstLimit: 5
+ rateLimit: 5
+ - paid:
+ quota:
+ limit: 5000
+ period: DAY
+ throttle:
+ burstLimit: 200
+ rateLimit: 100
I'll re-iterate the serverless documentation on setting up API
keys.
With this change, we can now limit by API keys, the number of requests that
come in on a set time period (in this case DAY
). Exceeding this limit will
result in a 429 Too Many Requests error.
$ curl 'https://3lskdjfsdlf8.execute-api.us-east-1.amazonaws.com/dev/hello' \
-H 'x-api-key: jsdlkfjslkj343434234lksjdflkj'
{"message":"Limit Exceeded"}
b8a7405
Integrate with DynamoDB
This is a big one. Also not included in this diff is package.json
and
package-lock.json
, needed because we introduced aws-sdk
as a dependency.
The two files can be created, however, with npm init --y && npm install
aws-sdk
.
To make things easier, let's walk through the changes per file.
diff --git a/blocks/lambda-webhook-callback/handler.js b/blocks/lambda-webhook-callback/handler.js
index b6e6b0d..956f2eb 100644
--- a/blocks/lambda-webhook-callback/handler.js
+++ b/blocks/lambda-webhook-callback/handler.js
@@ -1,11 +1,29 @@
+const AWS = require('aws-sdk')
+const dynamo = new AWS.DynamoDB.DocumentClient()
+
module.exports.hello = async event => {
let name = 'anonymous'
if (event.queryStringParameters && event.queryStringParameters.name) {
name = event.queryStringParameters.name
}
+ const params = {
+ TableName: process.env.TABLE_NAME
+ Item: {
+ name: name
+ }
+ }
+ const result = await dynamo.put(params).promise()
+
return {
statusCode: 200,
body: JSON.stringify(
We're using the aws-sdk to initialize a DynamoDB connection. DynamoDB is Amazon's own NoSQL offering and we'll be using it as a way to persist data between calls. At present, all we are saving to the table is the name of the user we're greeting.
Also note: for flexibility, we expect the table name to be available via the
TABLE_NAME
environment variable rather than hardcoding a value for it. The
reason why will become apparent when we look at the next file.
diff --git a/blocks/lambda-webhook-callback/serverless.yml b/blocks/lambda-webhook-callback/serverless.yml
index d889190..7e6fc28 100644
--- a/blocks/lambda-webhook-callback/serverless.yml
+++ b/blocks/lambda-webhook-callback/serverless.yml
@@ -46,6 +46,18 @@ provider:
throttle:
burstLimit: 200
rateLimit: 100
+ iamRoleStatements:
+ - Effect: Allow
+ Action:
+ - dynamodb:PutItem
+ Resource: !Sub 'arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${helloTable}'
+
+ environment:
+ TABLE_NAME: !Ref helloTable
@@ -58,3 +70,20 @@ functions:
method: get
private: true
cors: true
+
+resources:
+ Resources:
+ helloTable:
+ Type: AWS::DynamoDB::Table
+ Properties:
+ BillingMode: PAY_PER_REQUEST
+ AttributeDefinitions:
+ - AttributeName: name
+ AttributeType: S
+ KeySchema:
+ - AttributeName: name
+ KeyType: HASH
Let's look at the resources first. This section defines a DynamoDB table,
which we're referencing as helloTable
. The syntax for this section is
CloudFormation, as the serverless docs on
resources
suggest. As Dynamo is a NoSQL database, no schema is needed, but it does need
to know the table's primary key, which is name
in this case.
Up next, environment. The lambda will have access to these values through
the use of environment variables (eg: process.env.VARIABLE_NAME
). The !Ref
part is a bit confusing for someone not familiar with CloudFormation, so here
goes. In the previous section, helloTable
is the reference name for our
table, but it does not actually represent the name of the table when it gets
created. In the absense of a defined table name, CloudFormation will choose one
for us on stack create/update, so the only way to reference the name of our
table is through a Ref
value
If we created the database and our code knows the name of the database to
connect to, what else is left? iamRoleStatements, or access/authentication,
which is handled within AWS by IAM. IAM is based on the
RBAC access paradigm
. An IAM role can be configured to allow access to all sorts of resources
within AWS, but in this case the only action we need to do right now is a
PutItem
action on a DynamoDB. See IAM
docs.
The last potentially confusing part of the IAM section is the !Sub
resource
value. Functionally, this acts no differently than the !Ref
section of the
environments, with one difference. !Ref
only returns the name of the
resource, and for Resource
under IAM, we are required to provide the fully
qualified ARN value of the thing we're trying to access. It's a bit like
specifying the name of a file vs. specifying the full path to a file. More
documentation on the contents of the !Sub
section:
ARN,
Pseudo-parameter
references,
!Sub.
3f338a6
Count the number of visitors
diff --git a/blocks/lambda-webhook-callback/handler.js b/blocks/lambda-webhook-callback/handler.js
index e1b1496..6568753 100644
--- a/blocks/lambda-webhook-callback/handler.js
+++ b/blocks/lambda-webhook-callback/handler.js
@@ -14,20 +14,26 @@ module.exports.hello = async event => {
const params = {
TableName: process.env.TABLE_NAME,
- Item: {
- name: name,
- }
+ Key: {
+ name,
+ },
+ UpdateExpression: 'ADD #count :num',
+ ExpressionAttributeNames: {
+ '#count': 'greeting_count',
+ },
+ ExpressionAttributeValues: {
+ ':num': 1,
+ },
+ ReturnValues: 'UPDATED_NEW',
}
- const result = await dynamo.put(params).promise()
+ const result = await dynamo.update(params).promise()
return {
statusCode: 200,
body: JSON.stringify(
{
- message: `Hello ${name}!`
+ message: `Hello ${name} #${result.Attributes.greeting_count}!`
},
null,
2
diff --git a/blocks/lambda-webhook-callback/serverless.yml b/blocks/lambda-webhook-callback/serverless.yml
index 0741a40..d726cf7 100644
--- a/blocks/lambda-webhook-callback/serverless.yml
+++ b/blocks/lambda-webhook-callback/serverless.yml
@@ -53,7 +53,7 @@ provider:
iamRoleStatements:
- Effect: Allow
Action:
- - dynamodb:PutItem
+ - dynamodb:UpdateItem
Resource: !Sub 'arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${helloTable}'
With this commit, we change the response value of our endpoint to also include
the visitor count per name. Instead of putting an item, we are updating an item
(which can create new elements or update existing ones) and adding a
greeting_count
field (API docs on node.js DynamoDB
client).
Calling our API now looks like this:
$ curl 'https://34iisdflkj.execute-api.us-east-1.amazonaws.com/dev/hello' \
-H 'x-api-key: lksjdfljl3j45lk2345jlkjsdflksjdfljklsdkjf'
{
"message": "Hello anonymous #1!"
}
$ curl 'https://34iisdflkj.execute-api.us-east-1.amazonaws.com/dev/hello' \
-H 'x-api-key: lksjdfljl3j45lk2345jlkjsdflksjdfljklsdkjf'
{
"message": "Hello anonymous #2!"
}
$ curl 'https://34iisdflkj.execute-api.us-east-1.amazonaws.com/dev/hello?name=brooks' \
-H 'x-api-key: lksjdfljl3j45lk2345jlkjsdflksjdfljklsdkjf'
{
"message": "Hello brooks #1!"
}
6ddad1f
Dynamo: use SET, not ADD
diff --git a/blocks/lambda-webhook-callback/handler.js b/blocks/lambda-webhook-callback/handler.js
index 6568753..2e53aa4 100644
--- a/blocks/lambda-webhook-callback/handler.js
+++ b/blocks/lambda-webhook-callback/handler.js
@@ -17,11 +17,12 @@ module.exports.hello = async event => {
Key: {
name,
},
- UpdateExpression: 'ADD #count :num',
+ UpdateExpression: 'SET #count = if_not_exists(#count, :initial) + :num',
ExpressionAttributeNames: {
'#count': 'greeting_count',
},
ExpressionAttributeValues: {
+ ':initial': 0,
':num': 1,
},
ReturnValues: 'UPDATED_NEW',
This is a best practices change because functionally the two UpdateExpression
lines do the same thing. However, AWS recommends using SET in most
cases instead of ADD.
Initially I thought ADD would be a more obvious and simple solution, but the
more I think about it, SET makes it abundantly clear what the logic is if
greeting_count
does not yet exist as an attribute. ADD hides this logic, and
therefore has the potential to cause hard-to-debug errors.
4562185
Add DeletionPolicy note
--- a/blocks/lambda-webhook-callback/serverless.yml
+++ b/blocks/lambda-webhook-callback/serverless.yml
@@ -82,6 +82,12 @@ resources:
Resources:
helloTable:
Type: AWS::DynamoDB::Table
+ DeletionPolicy: Retain
Properties:
BillingMode: PAY_PER_REQUEST
AttributeDefinitions:
This change is optional, but in a production setting you most likely want to retain your database in the case of CloudFormation stack removal. The exception to this would be if your database contained only information that is easy to regenerate programmatically. For some resources, CloudFormation performs a backup before deletion, but in the case of DynamoDB, this does not happen.
Closing thoughts
And that's it.
This project only scratched the surface of what is possible with the right toolset and a healthy dose of documentation. In a real life scenario, a REST API server would be much more complicated not only in terms of supported features, but also in terms of team size and software process bottlenecks. But that's for you to figure out if you fit that scenario 🙃. Additional resources: