Increasing cloudwatch log groups subscription limit about elastic-serverless-forwarder HOT 5 CLOSED

My company is also running into this limitation. Support just advised that as it's a "known limitation" that it is not a bug. Development is advising to deploy directly with a bash script, but this is not a viable solution for our environment.

In case this helps someone else, though:

https://www.elastic.co/guide/en/esf/current/aws-deploy-elastic-serverless-forwarder.html#aws-serverless-forwarder-direct-deploy

from elastic-serverless-forwarder.

@whc4017 we are aware of this limit and we are going to state it clearly in the documentation and point to the alternative deployment solution we currently support, that's the one mentioned by @NYFreddie : https://www.elastic.co/guide/en/esf/current/aws-deploy-elastic-serverless-forwarder.html#aws-serverless-forwarder-direct-deploy

The limit is a mix of CloudFormation limits and relying on SAR for deployment, as well of the fact that we want to provide a deployment experience that not only deploys the ESF lambda and its components (like the continuing and replay queue) but also frees the users to manage on their own the triggers for ESF and the IAM permissions required.

In the specific context of SAR, in order to achieve that deployment UX, CloudFormation requires the usage of a Macro. Given the fact that Cloudformation does not support "dynamic" Macro reference, we are also consequentially limited to be able to have only one SAR deployment of ESF per region. That's exactly the reason for the failure of your 2nd approach.

Your 1st approach is currently discussed: as you experienced it will require two separated Parameters (something like ElasticServerlessForwarderCloudWatchLogsEvents and ElasticServerlessForwarderCloudWatchLogsEvents2).

This approach will work for a while, adding multiple separated Parameters, as much as needed to workaround the 4096 bytes limit multiplied by the total number of Parameters in order to fit the bytes size of the ARNs list you need to provide.
There's also a CloudFormation limit on the total number of Parameters that can be defined in a single template.

The above summarizes the reasons why, after a certain amount of AWS resources that users want to use as trigger of ESF, SAR deployment it is not the deployment solution they should rely on.
Again, we are aware this is currently not clear in ESF documentation, we apologize for that and we are going to address it.

Please note that the alternative deployment method that we offer and support, the direct one, while not based on SAR and not using CloudFormation Parameters, is still based on CloudFormation, and you may hit CloudFormation limits even using this solution. The main difference from SAR deployment is that you can have multiple deployments (since it does not use any Macro), so your 2nd approach, in the direct deployment context, is feasible.
To be a little more concrete: we expect that you will be able to need only one direct deployment if two separated Parameters would be enough for you with SAR deployment. But according on how much Parameters you currently need or will need in the future, you might need as well to split the ARNs list into multiple direct deployments.

from elastic-serverless-forwarder.

@NYFreddie thanks for chiming in and providing the link to the direct deployment solution

Please, see my comment above about the details of the current situation, both for SAR deployment and direct deployment.

We are aware that the direct deployment, while it overcomes the limit of SAR deployment, being a bash script is not a viable solution for every environment, like in your case.

We are already discussing about a pure terraform deployment solution, I cannot currently give you any ETA or expectation, but the value is clear and I will bring your case to discussion.

Thanks

from elastic-serverless-forwarder.

@whc4017 @NYFreddie

please find that v1.13.0 was released yesterday on SAR, including duplication of CF Parameters (#627)

This should unblock you for the moment

from elastic-serverless-forwarder.