• Description
• Deployment
• Test Drive
• FAQ
• License

• NPM needs to be installed
• AWS CLI needs to be installed and configured
• Serverless framework (https://serverless.com/)

Steps to deploy

1. Resolve NPM dependencies npm install
2. To customize your S3 bucket names, create a file "config.dev.yml" in the root directory, and add the following lines, replacing "IDENTIFIER" with a unique string:

s3ingestbucketname: edgesense-ingest-dev-IDENTIFIER
s3storebucketname: edgesense-store-dev-IDENTIFIER

3. Deploy resources on AWS sls deploy
4. Invoke the Init function to write test configuration and start the Kinesis analytics application sls invoke --function Init

This creates a CloudFormation stack with all necessary resources in your AWS account. To un-deploy, simply delete the stack.

1. Generate random test data. This simulates delivery of log data from Akamai DataStream node generate.js edgesense-ingest-dev-IDENTIFIER Note: replace IDENTIFIER with the unique string chosen for the ingestion S3 bucket above.
2. In CloudWatch, check the generated metrics in the "edgesense-dev" namespace that will start arriving after a couple of minutes
3. Create Athena tables and partitions, and use SQL to run queries on the log data (see Athena readme)
4. Customize the regular expressions used for URL pattern detection through DynamoDb, or use the import script:

cd Init
node importdata.js REGION DYNAMODB_TABLENAME CSVFILENAME

• The CloudWatch publishing function will eventually encounter rate-limiting issues in CloudWatch calls. Exponential back-off should be handled by AWS SDK; AWS support will need to be requested to raise rate limits if the rate of incoming data (and combinations of url pattern / response codes) is too high.

• Setup and Init
  • Give option to not create the S3 buckets in CloudFormation - most of the time, the buckets will have a longer lifecycle than the solution
• Ingest
  • Increase size of Kinesis records in PutRecordBatch to just below 5kb - Kinesis billing is based on the assumption that each record is equal to 5kb, using full 5kb would yield further cost savings
  • "WAFInfo" field needs to be configured in DataStream and passed through the pipe (not necessarily to collect metrics, rather to be able to run queries on it)
  • Tabs in URLs can still break parsing and lead to invalid values for certain fields
• Kinesis Analytics
  • Calculate anomaly scores (random forest cut trees)
• Monitoring
  • Automate creation of the a CloudWatch dashboard for monitoring of edgesense metrics
  • Automate creation of an example dashboard for generated metrics
  • Create SNS topic and alarms for errors in Lambda functions

See License