Comments (5)
My company is also running into this limitation. Support just advised that as it's a "known limitation" that it is not a bug. Development is advising to deploy directly with a bash script, but this is not a viable solution for our environment.
In case this helps someone else, though:
from elastic-serverless-forwarder.
@whc4017 we are aware of this limit and we are going to state it clearly in the documentation and point to the alternative deployment solution we currently support, that's the one mentioned by @NYFreddie : https://www.elastic.co/guide/en/esf/current/aws-deploy-elastic-serverless-forwarder.html#aws-serverless-forwarder-direct-deploy
The limit is a mix of CloudFormation limits and relying on SAR for deployment, as well of the fact that we want to provide a deployment experience that not only deploys the ESF lambda and its components (like the continuing and replay queue) but also frees the users to manage on their own the triggers for ESF and the IAM permissions required.
In the specific context of SAR, in order to achieve that deployment UX, CloudFormation requires the usage of a Macro. Given the fact that Cloudformation does not support "dynamic" Macro reference, we are also consequentially limited to be able to have only one SAR deployment of ESF per region. That's exactly the reason for the failure of your 2nd approach.
Your 1st approach is currently discussed: as you experienced it will require two separated Parameters (something like ElasticServerlessForwarderCloudWatchLogsEvents
and ElasticServerlessForwarderCloudWatchLogsEvents2
).
This approach will work for a while, adding multiple separated Parameters, as much as needed to workaround the 4096 bytes limit multiplied by the total number of Parameters in order to fit the bytes size of the ARNs list you need to provide.
There's also a CloudFormation limit on the total number of Parameters that can be defined in a single template.
The above summarizes the reasons why, after a certain amount of AWS resources that users want to use as trigger of ESF, SAR deployment it is not the deployment solution they should rely on.
Again, we are aware this is currently not clear in ESF documentation, we apologize for that and we are going to address it.
Please note that the alternative deployment method that we offer and support, the direct one, while not based on SAR and not using CloudFormation Parameters, is still based on CloudFormation, and you may hit CloudFormation limits even using this solution. The main difference from SAR deployment is that you can have multiple deployments (since it does not use any Macro), so your 2nd approach, in the direct deployment context, is feasible.
To be a little more concrete: we expect that you will be able to need only one direct deployment if two separated Parameters would be enough for you with SAR deployment. But according on how much Parameters you currently need or will need in the future, you might need as well to split the ARNs list into multiple direct deployments.
from elastic-serverless-forwarder.
@NYFreddie thanks for chiming in and providing the link to the direct deployment solution
Please, see my comment above about the details of the current situation, both for SAR deployment and direct deployment.
We are aware that the direct deployment, while it overcomes the limit of SAR deployment, being a bash script is not a viable solution for every environment, like in your case.
We are already discussing about a pure terraform deployment solution, I cannot currently give you any ETA or expectation, but the value is clear and I will bring your case to discussion.
Thanks
from elastic-serverless-forwarder.
please find that v1.13.0 was released yesterday on SAR, including duplication of CF Parameters (#627)
This should unblock you for the moment
from elastic-serverless-forwarder.
Related Issues (20)
- [Amazon Security Lake] Parquet File Support
- Docs about Internal retry queues (at least once delivery)
- Fix image for multiline documentation
- Include ESF specific User-Agent header in outgoing Elasticsearch requests
- Docs update need for getting esf running in vpc HOT 1
- Version Bump Automation HOT 1
- Add reserved concurrency to application properties
- Add ESF lambda ARN to SAR output
- permission for elastic-serverless-forwarder deployment user
- v1.11 missing outputs Lambda function ARN and IAM Role ARN HOT 2
- Invalid request provided: AWS::Logs::SubscriptionFilter HOT 4
- [DOCS] Document JSON content format we support
- [DOCS] Add missing documentation for `root_fields_to_add_to_expanded_event`
- Terraform definition HOT 1
- Test and support arm64
- Incorrect Handling of Permissions Can Result in Data Loss
- Provide support for Python 3.12
- [DOCS] Fix outdated link
- CloudWatch log retention for ESF own logs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elastic-serverless-forwarder.