Code Monkey home page Code Monkey logo

helix-coralogix-feeder's Introduction

Franklin Coralogix Feeder

Service that subscribes to CloudWatch logs for Franklin services and pushes them to Coralogix.

Status

codecov CircleCI GitHub license GitHub issues LGTM Code Quality Grade: JavaScript semantic-release

Installation

The AWS Console has an issue where a subscription filter can not be added with a specific version or alias, we therefore recommend to use the AWS CLI.

Given the service you want to push logs into Coralogix for, e.g. helix-services--my-service, use the following command:

$ aws logs put-subscription-filter \
  --log-group-name /aws/lambda/helix-services--my-service \
  --filter-name helix-coralogix-feeder \
  --filter-pattern '[timestamp=*Z, request_id="*-*", event]' \
  --destination-arn 'arn:aws:lambda:<region>:<accountid>:function:helix-services--coralogix-feeder:v1'

If you get an error that CloudWatch is not allowed to execute your function, add the following permission:

aws lambda add-permission \
    --function-name 'arn:aws:lambda:<region>:<accountid>:function:helix-services--coralogix-feeder:v1' \
    --statement-id 'CloudWatchInvokeCoralogixFeeder' \
    --principal 'logs.amazonaws.com' \
    --action 'lambda:InvokeFunction' \
    --source-arn 'arn:aws:logs:<region>:<accountId>:log-group:/aws/lambda/helix-services--my-service:*' \
    --source-account '<accountId>'

The service uses the following environment variables:

Name Description Required Default
CORALOGIX_API_KEY Coralogix Private Key Yes -
CORALOGIX_LOG_LEVEL Log level No info
CORALOGIX_SUBSYSTEM Subsystem No second segment in log group name, e.g. helix-services

If delivery to Coralogix fails, the service will send the unprocessed messages to the AWS SQS queue named helix-coralogix-feeder-dlq.

Development

Deploying Franklin Coralogix Feeder

All commits to main that pass the testing will be deployed automatically. All commits to branches that will pass the testing will get commited as helix-services--coralogix-feeder@ci<num> and tagged with the CI build number.

helix-coralogix-feeder's People

Contributors

dominique-pfister avatar renovate[bot] avatar rofe avatar semantic-release-bot avatar solaris007 avatar trieloff avatar tripodsan avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

helix-coralogix-feeder's Issues

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Warning

These dependencies are deprecated:

Datasource Name Replacement PR?
npm codecov Unavailable

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

github-actions
.github/workflows/main.yaml
  • actions/checkout v4
  • actions/setup-node v4
  • codecov/codecov-action v4
  • actions/checkout v4
  • actions/setup-node v4
  • aws-actions/configure-aws-credentials v4
  • aws-actions/aws-secretsmanager-get-secrets v2
  • actions/checkout v4
  • actions/setup-node v4
  • aws-actions/configure-aws-credentials v4
  • aws-actions/aws-secretsmanager-get-secrets v2
.github/workflows/semver-check.yaml
npm
package.json
  • @adobe/fetch 4.1.8
  • @adobe/helix-shared-wrap 2.0.2
  • @adobe/helix-status 10.1.2
  • aws4 1.13.1
  • @adobe/eslint-config-helix 2.0.6
  • @adobe/helix-deploy 12.0.8
  • @adobe/helix-universal 5.0.5
  • @adobe/semantic-release-coralogix 1.1.31
  • @semantic-release/changelog 6.0.3
  • @semantic-release/exec 6.0.3
  • @semantic-release/git 10.0.1
  • c8 10.1.2
  • codecov 3.8.3
  • dotenv 16.4.5
  • eslint 8.57.0
  • husky 9.1.4
  • junit-report-builder 4.0.1
  • lint-staged 15.2.8
  • mocha 10.7.3
  • mocha-multi-reporters 1.5.1
  • mocha-suppress-logs 0.5.1
  • nock 13.5.4
  • semantic-release 24.0.0

  • Check this box to trigger a request for Renovate to run again on this repository

Distinguish Subsystem By Lambda Alias

Is your feature request related to a problem? Please describe.
We'd like to distinguish between logs from lambdas we consider "development environment" versus from "production environments". For example, we'd like to apply the subsystem "my-service-dev" for lambdas invoked with the ci alias. Conversely, we'd like to apply the subsystem "my-service-prod" for lambdas invoked via the v1 lambda. This would allow for easy separation in Coralogix of source logs for viewing, alerting and graphing.

Describe the solution you'd like
On possible solution could be a feature that allows for mapping of subsystems based on lambda aliases via an env var CORALOGIX_ALIAS_MAPPING.

For example:

{
  "v1": "my-service-prod",
  "ci": "my-service-dev"
}

If no mapping is found, the feeder could continue to set the default subsystem as defined in the env var CORALOGIX_SUBSYSTEM.

Unprocessed log entries should be kept

From time to time, log entries are not accepted by coralogix (either connection reset or malformed). If this happens, log entries should be stored in a DLQ.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.