Comments (7)
@RaphaelAllier I opened a ticket with DD (internal ticket 1289207) on this, and we found that the CloudFormation template was not actually updating the Lambda code. The stack would change the Tag for the forwarder version, and did update the runtime. However in the logs, and in DD itself, the lambda reported as being on an old version.
I found that by running the cloudformation template update, then manually updating the lambda code with the datadog forwarder code, that it would work. But this meant that all future updates would need to be done this way. Which is odd.
My internal ticket closed because I did not respond due to being away. But basically we werent able to figure out why this happened.
from datadog-serverless-functions.
Hello,
I've opened a ticket to track this internally, in the meantime 3.73 should be stable
from datadog-serverless-functions.
Hello @kgochenour, when updating did the Layer version get updated as well ?
I redeployed the latest version from scratch using the CF template in our sandbox and didn't encounter this import issue.
EDIT: Installation from scratch worked but updating through CF with the latest.yaml
yields your issue, we're looking into it
from datadog-serverless-functions.
@kgochenour Hello. Following up as I was on PTO, we identified internally that there was indeed an issue with the Cloudformation update path. We're trying to pinpoint the issue and will post updates when we have a fix.
The issue seems to be located in the CF update path. Newer installations work fine with InstallAsLayer
set to true as the update only changes the layer (which contains the whole Forwarder code). However, older installations may see a conflict of dependencies that we're trying to investigate.
Thanks for the ticket number, I'll also have a look there
from datadog-serverless-functions.
After some troubleshooting I think we've narrowed down the issue here
To recap:
- Version 3.73.0 runs in Python 3.8
- Latest version (3.83.0) runs in Python 3.9
New installations of the Datadog Forwarder don't contain code in the Lambda itself, but in a layer that contains all the business logic. This is controlled in the Cloudformation template by the parameterInstallAsLayer
set totrue
by default. We recommend using this approach, although a non layered installation works just as well.
We tested various installations/setup and Cloudformation upgrade patterns.
- Using CF, deploy a 3.73.0 Forwarder with a non layered installation (Code is packaged in a .zip file on the Lambda). CF updates to the latest template work fine, and both 3.73.0 and latest version send logs to our intake.
- Using CF, deploy a 3.73.0 Forwarder with a layered installation (Code is packaged in a layer). CF updates to the latest template work fine, and both versions send logs to our intake
- Similar to what is described in the support ticket in this issue, we tried mixing Cloudformation updates with manual .zip code updates. Deploying a 3.73.0 Forwarder with the layer approach through CF and manually updating the .zip code to the latest version (picking a release .zip) will cause dependency conflicts to appear and show the error mentioned in this issue.
When the Forwarder is deployed with its code packaged in the layer, Cloudformation updates will only make a call to UpdateFunctionConfiguration
, which won't update the source code of the Lambda. If the function code has been manually updated, CF won't take that into account when the stack is re-applied, which may lead to errors.
Additionally, a workaround to sanitise the installation is to reapply the CF template twice, once with InstallAsLayer: false
and the second time with InstallAsLayer: true
, which should clean up the Lambda's source code.
Alternatively, fresh installation (not Cloudformation upgrades) of the latest version of the template are functional
from datadog-serverless-functions.
Hi @RaphaelAllier Do you have an upgrade guide? From reading the docs it looks like InstallAsLayer
defaults to true, but my reading of the post above suggests that it should be possible to upgrade with the following steps:
- deploy 3.73,
InstallAsLayer:true
(ie default) - deploy 3.73,
InstallAsLayer:false
- deploy 3.73,
InstallAsLayer:true
- deploy 3.91,
InstallAsLayer:true
Please can you confirm, or provide alternate instructions?
from datadog-serverless-functions.
Hello @anthonyangel
If your installation is currently using the version 3.73.0 and may have been manually upgraded in the past, steps to "sanitise" the installation (if needed, regular CF upgrades should work fine by default) should be:
Additional steps to make sure the forwarder code is completely packaged in the layer only, avoiding dependency conflicts
template URL: https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/3.73.0.yaml
- deploy same CF template version (3.73.0) with
InstallAsLayer: false
- deploy same CF template version (3.73.0) with
InstallAsLayer: true
From this step, your forwarder shouldn't have any code except in the Lambda layer attached to it (which is the standard way of deploying the Datadog Forwarder)
Regular upgrade process
template URL: https://datadog-cloudformation-template.s3.amazonaws.com/aws/forwarder/latest.yaml
- upgrade to the latest available CF template
To clarify, regular CF upgrades should always work fine. The errors listed in this issue we think resulted from a mix of CloudFormation upgrades and manual upgrades (like getting the last zip code from the latest release and manually installing it in the AWS console)
from datadog-serverless-functions.
Related Issues (20)
- Forwarder: Tagging logs with the proper `service` HOT 2
- [feature] have support for custom parsing additions to aws lambda function
- Lambda log forwarder - changed AWS Lambda enhanced metrics publish behaviour HOT 2
- Support docker image for log forwarder HOT 1
- Add region to cloudwatch (and others?) log collection
- Feature request: AWS Config support in AWS Forwarder Lambda
- Lambda layers fails (access denied) to deploy in AWS Gov Cloud after 3.77.0
- Missing tags for lambda@edge functions
- problem with api key length HOT 4
- Issue with role assignment on a management group (azure-integration/azuredeploy-managementgroups.json)
- No Python Version Matrix HOT 1
- Feature Request: Allow 'source' tag to be customized per AWS CloudWatch Log Group HOT 1
- Feature request: Allow option to dynamically set the 'Service' and 'Host' attributes from an AWS CloudWatch log group
- Activate Enhanced Metrics on Datadog Forwarder Lambda function
- Create Terraform module to support the deployment Datadog's Log Forwarder HOT 2
- Feature Request: Support reading API Gateway Access Logs from an S3 bucket HOT 2
- Azure Auditing logs to Datadog through EventGrid streamed incorrectly
- AWS Cloudformation configuration does not allow configuration of DD_LOG_LEVEL
- Feature Request: Datadog Forwarder Cloudformation - configurable Cloudwatch log group storage class
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datadog-serverless-functions.