Send ELB logs from S3 bucket to ElasticSearch using AWS Lambda

This directory contains terraform module to send ELB logs from S3 bucket to ElasticSearch using AWS Lambda

Particularly it creates:

1. Lambda function that does the sending
2. IAM role and policy that allows access to ES
3. S3 bucket notification that triggers the lambda function when an S3 object is created in the bucket.
4. (Only when your Lambda is deployed inside a VPC) Securitygroup for Lambda function

provider "aws" {
  region = "eu-central-1"
}

module "public_elb_logs_to_elasticsearch" {
  source        = "neillturner/elb-logs-to-elasticsearch/aws"
  version       = "0.1.0"

  prefix        = "public_es_"
  es_endpoint   = "test-es-XXXXXXX.eu-central-1.es.amazonaws.com"
  s3_bucket_arn = "arn:aws:s3:::XXXXXXX-elb-logs-eu-west-1"
  s3_bucket_id  = "XXXXXXX-elb-logs-eu-west-1"
}

module "vpc_elb_logs_to_elasticsearch" {
  source        = "neillturner/elb-logs-to-elasticsearch/aws"
  version       = "0.1.0"

  prefix        = "vpc_es_"
  es_endpoint   = "vpc-gc-demo-vpc-gloo5rzcdhyiykwdlots2hdjla.eu-central-1.es.amazonaws.com"
  s3_bucket_arn = "arn:aws:s3:::XXXXXXX-elb-logs-eu-west-1"
  s3_bucket_id  = "XXXXXXX-elb-logs-eu-west-1"
  subnet_ids    = ["subnet-d9990999"]
}

The zip file elb-logs-to-elasticsearch.zip was build from prithviraju/aws-elb-logs-to-elasticsearch

1. On your development machine, download and install Node.js.

2. Go to root folder of the repository and install node dependencies by running:

   npm install
   

   Verify that these are installed within the node_modules subdirectory.

3. To fix an issue with the module node-elb-log-parser causing errors

   TypeError: Cannot read property 'split' of undefined
   

   manually apply fix: TypeError: Cannot read property 'split' of undefined #6 #7

4. Create a zip file to package the index.js and the node_modules directory

   zip -r9 elb-logs-to-elasticsearch.zip *
   

The zip file thus created is the Lambda Deployment Package.