daniel-cottone / serverless-es-logs Goto Github PK
View Code? Open in Web Editor NEWA Serverless plugin to transport logs to ElasticSearch
License: MIT License
A Serverless plugin to transport logs to ElasticSearch
License: MIT License
๐ Problem
Currently the runtime of this plugin is hardcoded to nodejs12.x. The plugin should use the runtime defined in the service.
๐ฏ Goal
Allows others to configure the runtime for the lambda this log plugin will run on.
๐ก Possible Solutions
Use runtime from provider on serverless object
Is there a way to use an internal queue or something to make a smoother flow of data into ES to mitigate these errors
In index filed, can I have the variable date as we follow index pattern like this:
logs-2020.01.27
Where it follows pattern.
logs-YYYY.MM.DD
Is this one possible with this plugin?
Current status:
for the serverless project include more than 2 functions, after deploying, all functions are set up subscriptions. But some of them are not required to stream CloudWatch logs to ES.
Request Feature:
allow selecting a list of functions to apply the "serverless-es-logs"
Thanks
๐ Problem
When an AWS Elasticsearch cluster is configured to use a master-user, lambdas attempting to load data from CloudWatch need to authenticate.
๐ฏ Goal
It will help users keep their Elasticsearch clusters secure from unauthorized additions of data.
๐ก Possible Solutions
Perhaps in two additional arguments can be optionally passed to esLogs for the username and password. They can then be used to facilitate basic authentication.
Cloudformation supports expiration of Cloudwatch log groups: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-loggroup.html
We should add a configuration option to expire logs:
esLogs:
retentionInDays: 7
Sometimes events will have nested JSON that is stringify
'd at different levels, resulting in events with structure like
"name": "my-api",
"serviceName": "my-api",
"context": {
"callbackWaitsForEmptyEventLoop": true,
"functionVersion": "$LATEST",
"functionName": "some-function-name",
"memoryLimitInMB": "1024",
"logGroupName": "some-function-name",
"logStreamName": "2022/03/10/[$LATEST]8b6329fb30b54362a737fb013ffb3ee1",
"invokedFunctionArn": "arn:aws:lambda:us-east-1:PHONE_NUMBER:function:some-function-name",
"awsRequestId": "5556a991-4b86-4b8e-98a6-702f3871235f",
"invocationCount": 1,
"handlerType": "APIGateway",
"authorizer": {},
"correlationId": "0af3cf15-fceb-4402-ba0b-d2560fc3e3ad"
},
"hostname": "IP_ADDRESS",
"pid": 9,
"level": 30,
"restResponse": "{\"statusCode\":200,\"headers\":{\"Access-Control-Allow-Headers\":\"X-Requested-With, X-Api-Client, X-Api-Client-version\",\"Access-Control-Allow-Methods\":\"*\",\"Access-Control-Allow-Origin\":\"*\"},\"body\":{\"tracked\":true}}",
"msg": "Rest response info",
"time": "2022-03-10T18:02:26.394Z",
"src": {
"file": "/var/task/src/handler/MyFunction.js",
"line": 59,
"func": "info"
},
"v": 0
}
The inner JSON string (in restResponse
for this example) will not be decoded and split into fields in the document sent to elasticsearch.
๐ Problem
These inner JSON strings should be decoded into their own fields as well, to more robustly allow for log pipelines to feed through to ES in a format that is searchable.
๐ฏ Goal
This would extend Kibana capabilities with captured logs.
๐ก Possible Solutions
This could be resolved by attempting to JSON.parse
any non-numerical field in logsToEd.js:buildSource
. extractJson
already checks first to see if the first character is {
, so I imagine any performance impact from attempting to parse non-json strings would be minimal.
I ran into issue when setting up serverless-es-logs
for a service with ~50 endpoints:
The final policy size (20827) is bigger than the limit (20480). (Service: AWSLambda; Status Code: 400; Error Code: PolicyLengthExceededException)
The policy for EsLogsProcesserLambdaFunction
would go over the size limit because the plugin creates a single CWPermission
for each endpoint. As a quick and dirty fix for this I forked this repo (please ignore the shamefully committed dist
folder, when I say dirty I mean dirty) and commented out the subscription filter permission generation: https://github.com/immocapital/serverless-es-logs/blob/disable-individual-lambda-permissions/src/index.ts#L210-L228
Then, in my serverless.yml
, file I added
resources:
Resources:
LambdaCloudWatchPermissions:
Type: 'AWS::Lambda::Permission'
Properties:
Action: 'lambda:InvokeFunction'
SourceAccount: '#{AWS::AccountId}'
FunctionName:
"Fn::GetAtt":
- 'EsLogsProcesserNestedStack'
- 'Outputs.EsLogsProcesserLambdaFunctionArn'
Principal: 'logs.#{AWS::Region}.amazonaws.com'
SourceArn: 'arn:aws:logs:#{AWS::Region}:#{AWS::AccountId}:log-group:/aws/lambda/${self:service}-${self:custom.stage}-*'
DependsOn: 'EsLogsProcesserLambdaFunction'
(I use serverless-pseudo-parameters
and serverless-plugin-split-stacks
, just to explain those #{AWS...} variables and "Fn::GetAtt"
statement)
And this fixed the issue. This is by no means an ideal solution, so maybe adding an option to generate a single policy entry for the lambda would be a good idea.
Hi,
It looks like the plugin only supports AWS-hosted Elasticsearch: https://github.com/daniel-cottone/serverless-es-logs/blob/master/templates/code/logsToEs.js#L221
Are there any plans to support ElasticSearch in general (standalone or hosted elsewhere)? Our Elasticsearch is hosted outside of AWS.
How can we set the logFormat and remove the filterPattern?
As of 6.0, Elasticsearch does not support multiple values of _type
for a given index: https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html
This means the plugin is broken for any service that publishes logs from more than one source in its current state.
Updating filterPattern in the esLogs block of serverless.yml does not lead to any changes actually being made to the subscription filter.
๐ Problem
I was originally using the default filterPattern but now want to specify one. My assumption was that I could just update this in serverless.yml and I'd see the change reflected in the subscription filter for the log group.
๐ฏ Goal
Updating the subscription filter as needed when the esLogs configuration changes is expected Infrastructure-As-Code behavior. Thank you for taking a look!
_es-logs/index.js
9:5 error It's not necessary to initialize 'tags' to undefined no-undef-init
16:34 error A constructor name should not start with a lowercase letter new-cap
39:56 error Trailing spaces not allowed no-trailing-spaces
40:41 error Trailing spaces not allowed no-trailing-spaces
43:25 error Trailing spaces not allowed no-trailing-spaces
126:1 error Trailing spaces not allowed no-trailing-spaces
158:34 error Trailing spaces not allowed no-trailing-spaces
159:42 error Trailing spaces not allowed no-trailing-spaces
195:1 error Trailing spaces not allowed no-trailing-spaces
201:28 error Trailing spaces not allowed no-trailing-spaces
231:1 error Trailing spaces not allowed no-trailing-spaces
237:19 error Trailing spaces not allowed no-trailing-spaces
I'm trying to respond to the Serverless deprecation warning here by updating provider.lambdaHashingVersion
.
๐ Problem
I'm getting the following error when I try to deploy the updated function and I don't have a way to update the EsLogsProcesserLambda (which is generated automatically by this package).
An error occurred: EsLogsProcesserLambdaVersionHTahmkBFX9HLO6hMXGbu8Pdse32czw05FpX412PYF4 - A version for this Lambda function exists ( 3 ). Modify the function to create a new version..
๐ฏ Goal
Provide a way to change the es-logs-processor-lambda so that we can update the lambda hashing version and have a successful deployment.
I know this issue is not directly related to serverless-es-logs however you might be able to provide some guidance, I hope this can help other devs starting with the ELK stack (like myself).
We are sending logs using this plugin. Logs are collected with Winston, so they are mostly JSON objects with several properties.
Because we are using this plugin, all properties, in Kibana, are named something like e.g.: $event.statusCode
, at least if we use the default filter, which is fine.
Now while I'm able to create tables and some visualizations using those names, I can't create filters nor searches:
How do you usually handle this sort of scenario? I know this is a problem with Kibana itself, but I assume that you might have already lost some time investigating this so I hope you could share your findings!
Thank you! :)
๐ Problem
It looks like the plugin only supports AWS-hosted Elasticsearch: https://github.com/daniel-cottone/serverless-es-logs/blob/master/templates/code/logsToEs.js#L221
๐ฏ Goal
Our ElasticSearch is hosted on ElasticCloud and I would like to use this plugin to send logs to there
Hi,
I searched in the docs and issues to find the answer to my question, but I did not found what I was looking for, specifically is this log send to elastic search? From what I see it doesn't and I would like to plot this duration, mermory used etc to a dashboard.
REPORT RequestId: 4e352e1c-e997-4708-8a1e-eb40d9494be4 Duration: 1904.24 ms Billed Duration: 2000 ms Memory Size: 1024 MB Max Memory Used: 276 MB Init Duration: 2701.30 ms
I also see this errors in the logs of the logger (nice pun here)
INFO Failed Items: [ { "index": { "_index": "dev-logs-2019.10.31", "_type": "serverless-es-logs", "_id": "35068670517148905662465861963755689593093688542227529757", "status": 400, "error": { "type": "mapper_parsing_exception", "reason": "failed to parse field [@event.location] of type [text] in document with id '35068670517148905662465861963755689593093688542227529757'", "caused_by": { "type": "illegal_state_exception", "reason": "Can't get text on a START_OBJECT at 1:124" } } } }, ...]
I think that there is a mapping issue because in some logs there is a "location" property in a json object that has the type string (like "lat,lng") but on some other logs there is a location property like this "location": {lat: 42, lng: 42}
There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.
Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.
Hi Daniel,
Can we attach existing IAM role to es-logs lambda function?
Thanks.
Fixed in this PR: #313
Use custom deployment bucket, instead of default.
Would appreciate this feature, thank you in advance.
The variable jsonSubString
is not declared in the function buildSource
. This is throwing an error when using this plugin with serverless-webpack
which adds strict mode.
serverless-es-logs/templates/code/logsToEs.js
Line 111 in 283eb34
it seems there is a clash with serverless-warmup-plugin which causes the same index to be used
"errorMessage": "{\"statusCode\":200,\"responseBody\":\"{\\\"took\\\":26,\\\"errors\\\":true,\\\"items\\\":[{\\\"index\\\":{\\\"_index\\\":\\\"cwl-stock-middleware-production-2018.10.01\\\",\\\"_type\\\":\\\"/aws/lambda/stock-middleware-production-warmup-plugin\\\",\\\"_id\\\":\\\"34307478538089366886714326327315175922272989826627862528\\\",\\\"status\\\":400,\\\"error\\\":{\\\"type\\\":\\\"illegal_argument_exception\\\",\\\"reason\\\":\\\"Rejecting mapping update to [cwl-stock-middleware-production-2018.10.01] as the final mapping would have more than 1 type: [/aws/lambda/stock-middleware-production-kinesis, /aws/lambda/stock-middleware-production-warmup-plugin]\\\"}}}]}\"}"
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
Warning
These dependencies are deprecated:
Datasource | Name | Replacement PR? |
---|---|---|
npm | @types/faker |
|
npm | faker |
|
npm | tslint |
These updates are currently rate-limited. Click on a checkbox below to force their creation now.
faker
, @types/faker
)mocha
, @types/mocha
)@semantic-release/changelog
, @semantic-release/git
, @semantic-release/github
)fs-extra
, @types/fs-extra
)These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
sinon
, @types/sinon
)@semantic-release/git
, @semantic-release/github
)chai
, @types/chai
)mocha
, @types/mocha
).github/workflows/release.yml
actions/checkout v2
.github/workflows/status-check.yml
actions/checkout v2
actions/checkout v2
actions/checkout v2
actions/setup-node v2
package.json
fs-extra 9.0.0
lodash 4.17.21
@semantic-release/changelog 5.0.1
@semantic-release/git 9.0.0
@semantic-release/github 7.2.1
@types/chai 4.2.16
@types/faker 5.5.1
@types/fs-extra 8.1.1
@types/lodash 4.14.168
@types/mocha 8.2.2
@types/node 10.17.56
@types/sinon 10.0.0
chai 4.3.4
coveralls 3.1.0
faker 5.5.3
mocha 8.3.2
nyc 15.1.0
rimraf 3.0.2
semantic-release 17.4.2
sinon 10.0.0
ts-node 9.1.1
tslint 6.1.3
typescript 4.2.4
node >=10.0
.travis.yml
node 8
node 10
At the moment the way to enable API Gateway logs, using serverless-plugin-stage-variables is the working one (See this solution serverless/serverless#1918 (comment)).
The bug mentioned in the post is fixed and we have a valid Stage object in a resulting CloudFormation template.
But unfortunately the option includeApiGWLogs: true does not work in this case.
From what I see in the logs, this might be because ServerlessEsLogsPlugin.mergeCustomProviderResources()
is executed before Serverless: Merged stage variables into ApiGateway Deployment
.
Would be great if this would be possible to make them working together.
Since #535 this plugin can't be used with runtimes other than Node.js
๐ Problem
I've been using this plugin for a long time in several functions written in Python without any issues (good job by the way!!). But since the last version, the function is being created with Python as the runtime, and it obviously doesn't work
๐ฏ Goal
Restore the support for other runtimes than Node.js.
๐ก Possible Solutions
I don't see the point in inheriting the runtime, other than having to change Node version in two different places instead of one. Just reverting the change will be fine, however if the current behavior is valuable to someone, and you want to keep you can always check if the service runtime is Node.js first.
Hi!
I started to look at the code and I could not get to a conclusion about how the logs are forwarded to elastic search.
Are you using CloudWatch subscriptions feature, or, this plugin pushes each log manually to ES at runtime?
Thanks!
Hello, thank you for this wonderful plugin, the only issue I am facing right now is that I'm seeing the following error in my lambda.
DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
๐ Problem
Currently the lambdas created by the plugin are hardcoded to use 512 MB of memory at
serverless-es-logs/src/index.ts
Line 287 in ec7cb80
๐ฏ Goal
It would be great if the amount of memory could be configured. This would allow end-users more flexibility to support their use cases and control costs.
๐ก Possible Solutions
Currently AWS elasticsearch in a VPC configuration will not allow a remote account to post to it while VPC peering unless the lambda assumes a role in the remote account that has permissions to ESHttpPost.
๐ Problem
A simple way to address this would be to provide a methodology to inject credentials other than the current execution role into the Post call. This requires invoking STS AssumeRole to get new credentials against the targeted role arn and providing those to the Post call. This could be expanded upon to provide alternate posting methodologies to support self hosted elasticsearch but that would also require either a non-aws post function call with the necessary authorization headers/information or rework of the existing post function call to determine which authorization headers/information need to be sent.
๐ฏ Goal
Good security isolation would indicate to leverage multiple AWS accounts for serverless services. By separating out the accounts, you limit blast radius. However, in doing so you also want to have an account that can handle centralized log aggregation of some kind. Elasticsearch is a good target for that. This change would allow you to use the plugin as is, provide an environment variable for the role arn to assume, and provide for cross-account ESHttpPost calls.
๐ก Possible Solutions
#429 is a possible solution. I have tested and used it in the VPC elasticsearch configuration with vpc peering in place and assuming a role in the elasticsearch account which is separate from the serverless deployment account.
Plugin serverless-aws-alias creates Cloudwatch logs for API Gateway stages. We should add support for these log groups
An error occurred: EsLogsProcesserLambdaFunction - The provided execution role does not have permissions to call CreateNetworkInterface on EC2 (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: b9961307-18dd-11e9-980e-8b58ca234a14).
I am getting the following when trying to deploy? Any ideas
Hi,
I am witnessing this error which seems correlated to the fact that I am using the typescript plugin on my lambdas (I have other lambdas done in pure js and it's all good)
plugins:
- serverless-es-logs
- serverless-plugin-typescript
- serverless-offline
- serverless-domain-manager
- serverless-add-api-key
- serverless-iopipe-layers
that's my plugin section
Any clue what is happening here?
First, thank you for this very useful plugin!
It appears that the plugin does not support the case where the entire custom
YAML block is replaced by an external file. Given the below example:
serverless.yaml
custom: ${file(../../config/universal.yml):custom}
universal.yaml
custom:
esLogs:
endpoint: ${esEndpoint}
index: logs
retentionDays: 7
tags:
stack: ${opt:stage}
service: ${self:service}
Serverless deploy
throws ERROR: No configuration provided for serverless-es-logs!
. This is caused by destructuring inthe validatePluginOptions function.
I've been able to get the plugin to work in this scenario by replacing references to this.custom
with const { esLogs } = this.serverless.service.custom
after reading the following thread on the official discussion forum.
I'm unfamiliar with serverless plugins but would be happy to make a PR with this change if it's agreeable.
First of all thanks for building this. All my logs are now happily moving into my elasticsearch cluster :)
I had to fork this to make it work, I thought I would outline my changes, and why, and then you can see if you would be interested in me making a pull request
Right now, beyond a certain number of lambdas, then will fail to deploy, since the policy on the ingestion lambda becomes too big.
we have everything on 12.x, it would be nice to be able to configure the runtime version of the lambda deployed by this package.
why we are setting a node_options flag that is not available in node 10, so the lambda could not start,
I am not quite sure about this one, but I think this line https://github.com/daniel-cottone/serverless-es-logs/blob/master/templates/code/logsToEs.js#L124 is problematic. https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html
We could also allow an optional es version in the config to configure this. But I feel like at some point I will want to go in and modify that lambda anyway, so potentially giving users to define their own lambda might be a more flexible solution
Please let me know if you would like a pull request on any of the above. Again, thanks for putting this together :)
The Lambda IAM role doesn't attach to the es-logs-plugin
function if there is no provider role specified. Possible solution is to attach the IAM policy to the default role in this case
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.