This experiment simulates several calls to an API deployed through AWS CDK and the following services:

• Amazon API Gateway
• AWS Lambda
• Amazon DocumentDB
• AWS Secrets manager

The API is a url shortener. It stores the given url in body as json (see below for more details), associates it with a shortId and returns a shortUrl. If you access this shortUrl you will be redirected to the origin url.

Note: As the lambda function requires to be set into a VPC in order to connect to the DocumentDB database, our API will face more important cold start until the GA of the new AWS Lambda architecture that will share ENIs therefore we will implement a warm up phase in our service.

AWS CDK deploys the following architecture:

For this experiment you will need the following:

• The AWS CLI
• An AWS account. If you don’t have an AWS account, you can create a free account here.
• Node.js (>= 8.10). To install Node.js visit the node.js website. You can also a node version manager: nvm
• The AWS CDK toolkit: $> npm install -g aws-cdk
• artillery: $> npm install -g artillery

If this is the first time you deploy a CDK application in an AWS environment, you need to bootstrap it: cdk bootstrap. Please take a look at the bootstrap section of the CDK workshop.

This section covers how to deploy the API. It is a 3 steps process:

1. Creating secrets for the database
2. Building the application
3. Deploying the application

If you haven't done it already, execute the following command in your Terminal.

$> aws secretsmanager create-secret --name AwsExperimentsLoadApigwLambdaDocumentdb/docdbsecrets --secret-string '{"password": "XXX-XXX-XXX-XXXXXXXXXXXX","username": "masteruser"}'

Retrieve the arn for the displayed result, you will use it when you deploy the CDK application.

Note: we should be able to create the secret through CDK and suppress this manual step. Unfortunately, it didn't work for me. An issue is on its way to the CDK team.

At the root of the repository:

$> npm install

This will install all the AWS CDK project dependencies.

$> npm run clean

This will remove all eventually installed development dependencies in the API application code. This helps to reduce package size of our lambda functions.

$> npm run build

This command execute the following command:

• npm install --production --no-optional in the lambda-node directory to install required dependencies
• tsc in the root directory to transpile Typescript to Javascript
• wget https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem in the lambda-node to gather the necessary pem file for TLS connection to DocumentDB

$> cdk deploy --context docdbsecret_arn=arn:aws:secretsmanager:us-east-1:XXXXXXXXXXXX:secret:AwsExperimentsLoadApigwLambdaDocumentdb/docdbsecrets-XXXXXX -c docdbsecret_username=masteruser -c docdbsecret_password=XXX-XXX-XXX-XXX

The --context, -c option send parameters to the CDK application. Here you must replace XXXX values with the previously gathered arn and your desired password.

You can also specify the same context variables and values in the cdk.json file, use the following code:

{
  "app": "npx ts-node bin/aws-experiments-load-apigw-lambda-documentdb.ts",
  "context": {
    "docdbsecret_arn": "arn:aws:secretsmanager:us-east-1:XXXXXXXXXXXX:secret:AwsExperimentsLoadApigwLambdaDocumentdb/docdbsecrets-XXXXXX",
    "docdbsecret_username": "masteruser",
    "docdbsecret_password": "XXX-XXX-XXX-XXX"
  }
}

You can use curl to test your application manually.

For requesting a shorturl:

curl -d '{"url":"http://amazon.com"}' -H "Content-Type: application/json" -X POST https://XXXXXXXXXX.execute-api.us-east-1.amazonaws.com/prod/urls-node

For getting a complete url from a shorturl:

curl https://XXXXXXXXXX.execute-api.us-east-1.amazonaws.com/prod/urls-node/rBsbRVsjt

Or open the link in your browser.

Go to your load test folder:

$> cd tests/load

Edit the artillery.yml file to change the target field to match the output of the cdk deploy command.

Launch tests:

$> artillery run artillery.yml

The load test is based on the artillery script artillery.yml.

It sets two phases:

1. Warm up: ramp up arrival rate from 0 to 20 over 2 minutes
2. Max load sustain: 4 minutes at 20 virtual users arrival per second

Going further: if you run those tests from a local computer, you are limited by its bandwidth and performance. You can execute those tests from an EC2 instance in the VPC we have created to push the infrastructure further.

Navigate to the x-ray service map.

Tweak the selection to match the time you run the tests. You should see the following service map:

Click on the AWS::ApiGateway::Stage circle, click on "Analyze traces".

Zoom on the "Time series activity" diagram and click "Refine" in the "Filter trace set" card to zoom in. Repeat the operation if necessary.

You should now see something like this:

X-Ray detects that we hit the API 1102 times over a period of 6 minutes. Every time we got an HTTP 200 status code which means everything went well.

For further details on how to use X-Ray, please visit the documentation.

If you connect your DocumentDB cluster - you will need to launch a instance in one of the public subnet we created and connect to it in SSH - you'll be able to retrieve all the create shortened urls.

On the instance (after successfully connecting to the DocumentDB cluster):

rs0:PRIMARY> show dbs
staging  0.003GB
rs0:PRIMARY> use staging
switched to db staging
rs0:PRIMARY> show collections
urls
rs0:PRIMARY> db.urls.find()
{ "_id" : ObjectId("5d5d7a1ae6b4b300068e9365"), "url" : "https://www.amazon.com/", "createdAt" : "Wed, 21 Aug 2019 17:06:34 GMT", "shortId" : "jNW4wUmXa", "requesterIP" : "54.239.6.177" }
{ "_id" : ObjectId("5d5d7a1aae1025000682fe33"), "url" : "https://aws.amazon.com/", "createdAt" : "Wed, 21 Aug 2019 17:06:34 GMT", "shortId" : "XrMFf35UN", "requesterIP" : "54.239.6.177" }
...
Type "it" for more
rs0:PRIMARY> db.urls.count()
1217
rs0:PRIMARY> exit
bye
[ec2-user@ip-10-0-5-10 ~]$ exit
logout

Note: If the count() value differs from the x-ray results - like here - it may me because you have run manual or automatic tests earlier or that you x-ray selection doesn't account for all the executions.

Since this CDK project is typescript based, sources need to be compiled to JavaScript every time you make a modification to source files. This project is configured with a nice little npm script called watch that automatically compile .js file every time you make a change

In the home directory, open a new terminal and enter:

$> npm run watch

• npm run build compile typescript to js
• npm run watch watch for changes and compile
• cdk deploy deploy this stack to your default AWS account/region
• cdk diff compare deployed stack with current state
• cdk synth emits the synthesized CloudFormation template
• cdk destroy destroy the CDK application in your default AWS account/region

Destroy the CDK application:

$> cdk destroy

• Setting up X-RAY:
  • Documentation
  • CDK for API Gateway
  • CDK for Lambda
• Check lambda best practices and optimization:
  • Documentation
  • Optimizing AWS Lambda performance with MongoDB Atlas and Node.js
  • https://dev.to/adnanrahic/solving-invisible-scaling-issues-with-serverless-and-mongodb-4m55
  • How to Use MongoDB Connection Pooling on AWS Lambda
  • Best Practices for Developing on AWS Lambda
• Provide additional measuring results:
  • I’m afraid you’re thinking about AWS Lambda cold starts all wrong
  • aws lambda – compare cold start time with different languages, memory and code sizes
  • Lambda Cold Starts, A Language Comparison 🕵 ❄️
• Loading the application:
  • Artillery
  • Managing AWS Lambda Function Concurrency
• Create an issue for incapacity to create a document DB and secrets with AWS CDK
• Remove the hello world node and python
• Implement the lambda for the url shortener in python
• Implement the lambda for the url shortener in java
• Enhance tests
  • Implement a data generation phase to gather short urls
  • Implement a load test for getOriginURLNode
  • Implement a cleaning url that will delete all generated data to start over

This repository is inspired from different sources:

• Managing AWS Lambda Function Concurrency
• Connecting to AWS DocumentDB from a Lambda function
• Running AWS Lambda-based applications with Amazon DocumentDB
• For additional details on CDK, please take a look at the CDK workshop