Clarification request on memory/timeout for 'aws/logs_monitoring/lambda_function.py' about datadog-serverless-functions HOT 3 CLOSED

Hello @jalessio,

So indeed we should unified our recommendation across the different installation instructions.
I'll make sure that we do in the future.
That said, the memory and time used strongly depends of the type of data that are collected. Let me explain this.

Memory

It requires much more memory for ELB or ALB logs than for VPC or lambda logs.
Indeed ELB logs are created in batches every 5 minutes (in gzip files) whereas lambda logs are usually very small batches and do not consume much memory.
Therefore if you use ELB logs we recommend to set a higher memory limit.

Timeout

The lambda calls should almost never take more than a couple seconds, that said in case of network issues, it can be important to leave some time to the lambda to retry otherwise the logs cannot be sent.

Summary

Considering the above, you have to options:

• Optimised: If you do not have ELB logs: 124MB and 10 seconds should be fine
• Safe: with ELB logs, looking at the memory used by the lambda to send those would give you a good approximate of the required memory to forward those logs safely. I would recommend 512MB and 120seconds (but that depends obviously of the size of your ELB log batches).

Let me know if that clarifies the different options that you have.

from datadog-serverless-functions.

Thanks for the clarification! For ELB logs I guess we will monitor the memory usage of our lambda jobs and adjust accordingly but it's great to have reasonable starting point.

from datadog-serverless-functions.

Thanks for the feedback. I'll close this issue then.

The requirements to update the documentation to have it sync is tracked on our side and will be done this week.

Thanks again for helping us to improve the Datadog product.

from datadog-serverless-functions.