Code Monkey home page Code Monkey logo

aws-centralize-logs-using-cdk's Introduction

Centralize CloudWatch Logs with CDK

Solution

This solution mainly involves Amazon Kinesis Data Firehose which provides the ability to process data in real-time allowing critical use cases to be implemented based on it. The centralized logging account will expose a Log Destination endpoint which in turn is connected to a Kinesis Firehose. Kinesis Firehose is configured to push data to Amazon S3 Bucket. We could configure a lambda to un-compress/format the data before it is sent to S3. Also we can utilize Kinesis Firehose configuration to transform the data using AWS Glue before pushing it. Also, Firehose can not only push the data to S3 but also supports other destinations like Amazon Redshift, Amazon Elasticsearch Service and Splunk.

Alt text

Note: Though the diagram shows two different accounts, this solution can be deployed in one single account as well.

The AWS Cloud Development Kit (AWS CDK) is an open source software development framework to model and provision your cloud application resources using familiar programming languages. CDK with C# has been used in this solution to create our Infrastructure as Code.

Pre-Requisites

  • AWS CDK [Version 1.36]
  • .Net Core 3.1

Commands to Deploy the Solution

  • git clone https://github.com/navbalaraman/central-logs-cdk.git
  • cd central-logs-cdk
  • dotnet build src

If deploying the solution to a single account:

Step 1: Bootstrap your account to prepare the environment with resources that will be required for the deployment.

  • cdk bootstrap

Step 2: Deploy the LogDestinationStack (Replace AWS-ACCOUNT-ID with your AWS Account number)

  • cdk deploy LogDestinationStack --parameters LogDestinationStack:SourceAccountNumber="AWS-ACCOUNT-ID"

Alt text

Step 3: Deploy the LogSourceStack (Replace LOG-DESTINATION-ARN with the output value from the previous command, and CLOUDWATCH-LOGGROUP with the name of the Log group)

  • cdk deploy LogSourceStack --parameters LogSourceStack:LogGroupName="CLOUDWATCH-LOGGROUP" --parameters LogDestinationArn="LOG-DESTINATION-ARN"

If deploying the solution to separate source and destination account:

Step 1: Deploy the LogsDestinationStack (Replace SOURCE-AWS-ACCOUNT-ID with your AWS Account number)

  • cdk bootstrap
  • cdk deploy LogDestinationStack --parameters LogDestinationStack:SourceAccountNumber="SOURCE-AWS-ACCOUNT-ID"

Alt text

Step 2: Deploy the LogSourceStack (Replace LOG-DESTINATION-ARN with the output value from the previous command, and CLOUDWATCH-LOGGROUP with the name of the Log group)

  • cdk bootstrap
  • cdk deploy LogSourceStack --parameters LogSourceStack:LogGroupName="CLOUDWATCH-LOGGROUP" --parameters LogDestinationArn="LOG-DESTINATION-ARN"

aws-centralize-logs-using-cdk's People

Contributors

amazon-auto avatar navbalaraman avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

mikelane arpowers mnanchev markshuang itwiths stohlthomas

aws-centralize-logs-using-cdk's Issues

Deploy source gives RoleArn error

When attempting to deploy the LogSourceStack, I get the following error:

3:23:25 PM | CREATE_FAILED        | 
AWS::Logs::SubscriptionFilter 
SubscriptionFilter
destinationArn for vendor cloudformation cannot be used without roleArn (Service: AWSLogs; Status Code: 400; 
Error Code:InvalidParameterException; Request ID: db0...

This error repeats 4 times.
We created all role types and principal types in an attempt to work around this.
When running as a role we get a different error:

3:16:17 PM | CREATE_FAILED        
WS::Logs::SubscriptionFilter 
SubscriptionFilter
PutSubscriptionFilter operation cannot work with destinationArn for vendor cloudformation 
(Service: AWSLogs; Status Code: 400; Error Code: InvalidParameterException; 
Request ID: ...

Ideas?

The documentation suggest that the cross account destinations only work with Kinesis, so I am confused how this is supposed to work.

Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.