daniel-cottone / serverless-es-logs Goto Github PK

🐞 Problem

Currently the runtime of this plugin is hardcoded to nodejs12.x. The plugin should use the runtime defined in the service.

🎯 Goal

Allows others to configure the runtime for the lambda this log plugin will run on.

💡 Possible Solutions

Use runtime from provider on serverless object

Is there a way to use an internal queue or something to make a smoother flow of data into ES to mitigate these errors

In index filed, can I have the variable date as we follow index pattern like this:
logs-2020.01.27
Where it follows pattern.
logs-YYYY.MM.DD
Is this one possible with this plugin?

Current status:
for the serverless project include more than 2 functions, after deploying, all functions are set up subscriptions. But some of them are not required to stream CloudWatch logs to ES.

Request Feature:
allow selecting a list of functions to apply the "serverless-es-logs"

Thanks

🐞 Problem
When an AWS Elasticsearch cluster is configured to use a master-user, lambdas attempting to load data from CloudWatch need to authenticate.

🎯 Goal
It will help users keep their Elasticsearch clusters secure from unauthorized additions of data.

💡 Possible Solutions
Perhaps in two additional arguments can be optionally passed to esLogs for the username and password. They can then be used to facilitate basic authentication.

Cloudformation supports expiration of Cloudwatch log groups: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-loggroup.html

We should add a configuration option to expire logs:

esLogs:
  retentionInDays: 7

Sometimes events will have nested JSON that is stringify'd at different levels, resulting in events with structure like

  "name": "my-api",
  "serviceName": "my-api",
  "context": {
    "callbackWaitsForEmptyEventLoop": true,
    "functionVersion": "$LATEST",
    "functionName": "some-function-name",
    "memoryLimitInMB": "1024",
    "logGroupName": "some-function-name",
    "logStreamName": "2022/03/10/[$LATEST]8b6329fb30b54362a737fb013ffb3ee1",
    "invokedFunctionArn": "arn:aws:lambda:us-east-1:PHONE_NUMBER:function:some-function-name",
    "awsRequestId": "5556a991-4b86-4b8e-98a6-702f3871235f",
    "invocationCount": 1,
    "handlerType": "APIGateway",
    "authorizer": {},
    "correlationId": "0af3cf15-fceb-4402-ba0b-d2560fc3e3ad"
  },
  "hostname": "IP_ADDRESS",
  "pid": 9,
  "level": 30,
  "restResponse": "{\"statusCode\":200,\"headers\":{\"Access-Control-Allow-Headers\":\"X-Requested-With, X-Api-Client, X-Api-Client-version\",\"Access-Control-Allow-Methods\":\"*\",\"Access-Control-Allow-Origin\":\"*\"},\"body\":{\"tracked\":true}}",
  "msg": "Rest response info",
  "time": "2022-03-10T18:02:26.394Z",
  "src": {
    "file": "/var/task/src/handler/MyFunction.js",
    "line": 59,
    "func": "info"
  },
  "v": 0
}

The inner JSON string (in restResponse for this example) will not be decoded and split into fields in the document sent to elasticsearch.

🐞 Problem

These inner JSON strings should be decoded into their own fields as well, to more robustly allow for log pipelines to feed through to ES in a format that is searchable.

🎯 Goal

This would extend Kibana capabilities with captured logs.

💡 Possible Solutions

This could be resolved by attempting to JSON.parse any non-numerical field in logsToEd.js:buildSource. extractJson already checks first to see if the first character is {, so I imagine any performance impact from attempting to parse non-json strings would be minimal.

I ran into issue when setting up serverless-es-logs for a service with ~50 endpoints:

The final policy size (20827) is bigger than the limit (20480). (Service: AWSLambda; Status Code: 400; Error Code: PolicyLengthExceededException)

The policy for EsLogsProcesserLambdaFunction would go over the size limit because the plugin creates a single CWPermission for each endpoint. As a quick and dirty fix for this I forked this repo (please ignore the shamefully committed dist folder, when I say dirty I mean dirty) and commented out the subscription filter permission generation: https://github.com/immocapital/serverless-es-logs/blob/disable-individual-lambda-permissions/src/index.ts#L210-L228

Then, in my serverless.yml, file I added

resources:
  Resources:
    LambdaCloudWatchPermissions:
      Type: 'AWS::Lambda::Permission'
      Properties:
        Action: 'lambda:InvokeFunction'
        SourceAccount: '#{AWS::AccountId}'
        FunctionName:
          "Fn::GetAtt":
            - 'EsLogsProcesserNestedStack'
            - 'Outputs.EsLogsProcesserLambdaFunctionArn'
        Principal: 'logs.#{AWS::Region}.amazonaws.com'
        SourceArn: 'arn:aws:logs:#{AWS::Region}:#{AWS::AccountId}:log-group:/aws/lambda/${self:service}-${self:custom.stage}-*'
      DependsOn: 'EsLogsProcesserLambdaFunction'

(I use serverless-pseudo-parameters and serverless-plugin-split-stacks, just to explain those #{AWS...} variables and "Fn::GetAtt" statement)

And this fixed the issue. This is by no means an ideal solution, so maybe adding an option to generate a single policy entry for the lambda would be a good idea.

Hi,
It looks like the plugin only supports AWS-hosted Elasticsearch: https://github.com/daniel-cottone/serverless-es-logs/blob/master/templates/code/logsToEs.js#L221

Are there any plans to support ElasticSearch in general (standalone or hosted elsewhere)? Our Elasticsearch is hosted outside of AWS.

How can we set the logFormat and remove the filterPattern?

As of 6.0, Elasticsearch does not support multiple values of _type for a given index: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html

This means the plugin is broken for any service that publishes logs from more than one source in its current state.

Updating filterPattern in the esLogs block of serverless.yml does not lead to any changes actually being made to the subscription filter.

🐞 Problem
I was originally using the default filterPattern but now want to specify one. My assumption was that I could just update this in serverless.yml and I'd see the change reflected in the subscription filter for the log group.

🎯 Goal
Updating the subscription filter as needed when the esLogs configuration changes is expected Infrastructure-As-Code behavior. Thank you for taking a look!

_es-logs/index.js
9:5 error It's not necessary to initialize 'tags' to undefined no-undef-init
16:34 error A constructor name should not start with a lowercase letter new-cap
39:56 error Trailing spaces not allowed no-trailing-spaces
40:41 error Trailing spaces not allowed no-trailing-spaces
43:25 error Trailing spaces not allowed no-trailing-spaces
126:1 error Trailing spaces not allowed no-trailing-spaces
158:34 error Trailing spaces not allowed no-trailing-spaces
159:42 error Trailing spaces not allowed no-trailing-spaces
195:1 error Trailing spaces not allowed no-trailing-spaces
201:28 error Trailing spaces not allowed no-trailing-spaces
231:1 error Trailing spaces not allowed no-trailing-spaces
237:19 error Trailing spaces not allowed no-trailing-spaces

I'm trying to respond to the Serverless deprecation warning here by updating provider.lambdaHashingVersion.

🐞 Problem
I'm getting the following error when I try to deploy the updated function and I don't have a way to update the EsLogsProcesserLambda (which is generated automatically by this package).

An error occurred: EsLogsProcesserLambdaVersionHTahmkBFX9HLO6hMXGbu8Pdse32czw05FpX412PYF4 - A version for this Lambda function exists ( 3 ). Modify the function to create a new version..

🎯 Goal
Provide a way to change the es-logs-processor-lambda so that we can update the lambda hashing version and have a successful deployment.

I know this issue is not directly related to serverless-es-logs however you might be able to provide some guidance, I hope this can help other devs starting with the ELK stack (like myself).

We are sending logs using this plugin. Logs are collected with Winston, so they are mostly JSON objects with several properties.

Because we are using this plugin, all properties, in Kibana, are named something like e.g.: $event.statusCode , at least if we use the default filter, which is fine.

Now while I'm able to create tables and some visualizations using those names, I can't create filters nor searches:

How do you usually handle this sort of scenario? I know this is a problem with Kibana itself, but I assume that you might have already lost some time investigating this so I hope you could share your findings!

Thank you! :)

🐞 Problem
It looks like the plugin only supports AWS-hosted Elasticsearch: https://github.com/daniel-cottone/serverless-es-logs/blob/master/templates/code/logsToEs.js#L221

🎯 Goal
Our ElasticSearch is hosted on ElasticCloud and I would like to use this plugin to send logs to there

Hi,

I searched in the docs and issues to find the answer to my question, but I did not found what I was looking for, specifically is this log send to elastic search? From what I see it doesn't and I would like to plot this duration, mermory used etc to a dashboard.
REPORT RequestId: 4e352e1c-e997-4708-8a1e-eb40d9494be4 Duration: 1904.24 ms Billed Duration: 2000 ms Memory Size: 1024 MB Max Memory Used: 276 MB Init Duration: 2701.30 ms

I also see this errors in the logs of the logger (nice pun here)

INFO Failed Items: [ { "index": { "_index": "dev-logs-2019.10.31", "_type": "serverless-es-logs", "_id": "35068670517148905662465861963755689593093688542227529757", "status": 400, "error": { "type": "mapper_parsing_exception", "reason": "failed to parse field [@event.location] of type [text] in document with id '35068670517148905662465861963755689593093688542227529757'", "caused_by": { "type": "illegal_state_exception", "reason": "Can't get text on a START_OBJECT at 1:124" } } } }, ...]

I think that there is a mapping issue because in some logs there is a "location" property in a json object that has the type string (like "lat,lng") but on some other logs there is a location property like this "location": {lat: 42, lng: 42}

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

Hi Daniel,

Can we attach existing IAM role to es-logs lambda function?

Thanks.

Fixed in this PR: #313

Use custom deployment bucket, instead of default.
Would appreciate this feature, thank you in advance.

The variable jsonSubString is not declared in the function buildSource. This is throwing an error when using this plugin with serverless-webpack which adds strict mode.

it seems there is a clash with serverless-warmup-plugin which causes the same index to be used

"errorMessage": "{\"statusCode\":200,\"responseBody\":\"{\\\"took\\\":26,\\\"errors\\\":true,\\\"items\\\":[{\\\"index\\\":{\\\"_index\\\":\\\"cwl-stock-middleware-production-2018.10.01\\\",\\\"_type\\\":\\\"/aws/lambda/stock-middleware-production-warmup-plugin\\\",\\\"_id\\\":\\\"34307478538089366886714326327315175922272989826627862528\\\",\\\"status\\\":400,\\\"error\\\":{\\\"type\\\":\\\"illegal_argument_exception\\\",\\\"reason\\\":\\\"Rejecting mapping update to [cwl-stock-middleware-production-2018.10.01] as the final mapping would have more than 1 type: [/aws/lambda/stock-middleware-production-kinesis, /aws/lambda/stock-middleware-production-warmup-plugin]\\\"}}}]}\"}"

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Datasource	Name	Replacement PR?
npm	@types/faker
npm	faker
npm	tslint

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• chore(deps): replace dependency faker with @faker-js/faker
• chore(deps): update actions/checkout action to v4
• chore(deps): update actions/setup-node action to v4
• chore(deps): update dependency @types/node to v20
• chore(deps): update dependency @types/sinon to v17
• chore(deps): update dependency chai to v5
• chore(deps): update dependency faker to v6 (faker, @types/faker)
• chore(deps): update dependency mocha to v10 (mocha, @types/mocha)
• chore(deps): update dependency nyc to v17
• chore(deps): update dependency rimraf to v6
• chore(deps): update dependency sinon to v18
• chore(deps): update dependency typescript to v5
• chore(deps): update semantic-release monorepo (major) (@semantic-release/changelog, @semantic-release/git, @semantic-release/github)
• fix(deps): update dependency fs-extra to v11 (fs-extra, @types/fs-extra)
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency semantic-release to v19 [security]
• chore(deps): update dependency @types/faker to v5.5.9
• chore(deps): update dependency @types/fs-extra to v8.1.5
• chore(deps): update dependency @types/node to v10.17.60
• chore(deps): update dependency coveralls to v3.1.1
• chore(deps): update dependency sinon to v10.0.1 (sinon, @types/sinon)
• chore(deps): update semantic-release monorepo (@semantic-release/git, @semantic-release/github)
• chore(deps): update dependency @types/lodash to v4.17.7
• chore(deps): update dependency chai to v4.5.0 (chai, @types/chai)
• chore(deps): update dependency mocha to v8.4.0 (mocha, @types/mocha)
• chore(deps): update dependency typescript to v4.9.5
• fix(deps): update dependency fs-extra to v9.1.0
• chore(deps): update dependency ts-node to v10
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

At the moment the way to enable API Gateway logs, using serverless-plugin-stage-variables is the working one (See this solution serverless/serverless#1918 (comment)).
The bug mentioned in the post is fixed and we have a valid Stage object in a resulting CloudFormation template.

But unfortunately the option includeApiGWLogs: true does not work in this case.
From what I see in the logs, this might be because ServerlessEsLogsPlugin.mergeCustomProviderResources() is executed before Serverless: Merged stage variables into ApiGateway Deployment.

Would be great if this would be possible to make them working together.

Since #535 this plugin can't be used with runtimes other than Node.js

🐞 Problem
I've been using this plugin for a long time in several functions written in Python without any issues (good job by the way!!). But since the last version, the function is being created with Python as the runtime, and it obviously doesn't work

🎯 Goal
Restore the support for other runtimes than Node.js.

💡 Possible Solutions
I don't see the point in inheriting the runtime, other than having to change Node version in two different places instead of one. Just reverting the change will be fine, however if the current behavior is valuable to someone, and you want to keep you can always check if the service runtime is Node.js first.

Hi!
I started to look at the code and I could not get to a conclusion about how the logs are forwarded to elastic search.

Are you using CloudWatch subscriptions feature, or, this plugin pushes each log manually to ES at runtime?

Thanks!

Hello, thank you for this wonderful plugin, the only issue I am facing right now is that I'm seeing the following error in my lambda.

DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.

🐞 Problem
Currently the lambdas created by the plugin are hardcoded to use 512 MB of memory at

🎯 Goal
It would be great if the amount of memory could be configured. This would allow end-users more flexibility to support their use cases and control costs.

💡 Possible Solutions

Currently AWS elasticsearch in a VPC configuration will not allow a remote account to post to it while VPC peering unless the lambda assumes a role in the remote account that has permissions to ESHttpPost.

🐞 Problem

A simple way to address this would be to provide a methodology to inject credentials other than the current execution role into the Post call. This requires invoking STS AssumeRole to get new credentials against the targeted role arn and providing those to the Post call. This could be expanded upon to provide alternate posting methodologies to support self hosted elasticsearch but that would also require either a non-aws post function call with the necessary authorization headers/information or rework of the existing post function call to determine which authorization headers/information need to be sent.

🎯 Goal

Good security isolation would indicate to leverage multiple AWS accounts for serverless services. By separating out the accounts, you limit blast radius. However, in doing so you also want to have an account that can handle centralized log aggregation of some kind. Elasticsearch is a good target for that. This change would allow you to use the plugin as is, provide an environment variable for the role arn to assume, and provide for cross-account ESHttpPost calls.

💡 Possible Solutions

#429 is a possible solution. I have tested and used it in the VPC elasticsearch configuration with vpc peering in place and assuming a role in the elasticsearch account which is separate from the serverless deployment account.

Plugin serverless-aws-alias creates Cloudwatch logs for API Gateway stages. We should add support for these log groups

An error occurred: EsLogsProcesserLambdaFunction - The provided execution role does not have permissions to call CreateNetworkInterface on EC2 (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: b9961307-18dd-11e9-980e-8b58ca234a14).

I am getting the following when trying to deploy? Any ideas

Hi,
I am witnessing this error which seems correlated to the fact that I am using the typescript plugin on my lambdas (I have other lambdas done in pure js and it's all good)

plugins:
  - serverless-es-logs
  - serverless-plugin-typescript
  - serverless-offline
  - serverless-domain-manager
  - serverless-add-api-key
  - serverless-iopipe-layers

that's my plugin section

Any clue what is happening here?

First, thank you for this very useful plugin!

It appears that the plugin does not support the case where the entire custom YAML block is replaced by an external file. Given the below example:

serverless.yaml
custom: ${file(../../config/universal.yml):custom}

universal.yaml
custom:
  esLogs:
    endpoint: ${esEndpoint}
    index: logs
    retentionDays: 7
    tags:
      stack: ${opt:stage}
      service: ${self:service}

Serverless deploy throws ERROR: No configuration provided for serverless-es-logs!. This is caused by destructuring inthe validatePluginOptions function.

I've been able to get the plugin to work in this scenario by replacing references to this.custom with const { esLogs } = this.serverless.service.custom after reading the following thread on the official discussion forum.

I'm unfamiliar with serverless plugins but would be happy to make a PR with this change if it's agreeable.

First of all thanks for building this. All my logs are now happily moving into my elasticsearch cluster :)

I had to fork this to make it work, I thought I would outline my changes, and why, and then you can see if you would be interested in me making a pull request

Problem 1 - policy size

Right now, beyond a certain number of lambdas, then will fail to deploy, since the policy on the ingestion lambda becomes too big.

• my solution - https://github.com/edyncare/serverless-es-logs/blob/master/src/index.ts#L252-L266 , I wildcard the arn
• potential solution for serverless-es-logs - I do not have any experience with serverless, so I am not sure if all the lambdas deployed in a particular stage have a common prefix, or if we can get the prefix, but if it is possible that would be my suggestion

Problem 2 - lambda runtime version

we have everything on 12.x, it would be nice to be able to configure the runtime version of the lambda deployed by this package.
why we are setting a node_options flag that is not available in node 10, so the lambda could not start,

Problem 3 - incompatible with latest elastic search

I am not quite sure about this one, but I think this line https://github.com/daniel-cottone/serverless-es-logs/blob/master/templates/code/logsToEs.js#L124 is problematic. https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html
We could also allow an optional es version in the config to configure this. But I feel like at some point I will want to go in and modify that lambda anyway, so potentially giving users to define their own lambda might be a more flexible solution

Please let me know if you would like a pull request on any of the above. Again, thanks for putting this together :)

The Lambda IAM role doesn't attach to the es-logs-plugin function if there is no provider role specified. Possible solution is to attach the IAM policy to the default role in this case