datadog / datadog-lambda-go Goto Github PK

The Datadog AWS Lambda package for Go

License: Apache License 2.0

Go 92.27% Shell 7.50% JavaScript 0.23%

datadog-lambda-go's Introduction

datadog-lambda-go

Datadog Lambda Library for Go enables enhanced Lambda metrics, distributed tracing, and custom metric submission from AWS Lambda functions.

Installation

Follow the installation instructions, and view your function's enhanced metrics, traces and logs in Datadog.

Configurations

See the advanced configuration options to tag your telemetry, capture request/response payloads, filter or scrub sensitive information from logs or traces, and more.

Opening Issues

If you encounter a bug with this package, we want to hear about it. Before opening a new issue, search the existing issues to avoid duplicates.

When opening an issue, include the datadog-lambda-go version, go version, and stack trace if available. In addition, include the steps to reproduce when appropriate.

You can also open an issue for a feature request.

Contributing

If you find an issue with this package and have a fix, please feel free to open a pull request following the procedures.

Community

For product feedback and questions, join the #serverless channel in the Datadog community on Slack.

License

Unless explicitly stated otherwise all files in this repository are licensed under the Apache License Version 2.0.

datadog-lambda-go's People

Contributors

Stargazers

Watchers

datadog-lambda-go's Issues

Child spans aren't displayed when using tracer.WithErr(err) to add error data

Apologies this one is slightly difficult to explain - please let me know if you need more information.

Expected Behavior

When setting errors on child spans, I should be able to see these child span errors within the full trace in the Datadog console.

Flame graph:

->  aws.lambda foo.service
            --> Initialization
                               --> aws.lambda.foobar foo.service
                                    --> Bar package.Foo (error)

Actual Behavior

So throughout our applications we use the following pattern:

func (f *Foo) Bar(ctx context.Context, ...) (err error) {
	span, spanCtx := tracer.StartSpanFromContext(ctx, "Bar", tracer.ResourceName("package.Foo"))
	defer func() {
		span.Finish(tracer.WithError(err))
	}()

	...

	return nil
}

This works perfectly in our containerized applications. We get a nice trace hierarchy and can see exactly which child traces have returned errorrs in the full trace.

As soon as we use this in a Lambda the child traces aren't displayed in the trace anymore. All we can see is the spans created by this Datadog library and none of our own.

Instead of seeing the expected behavior flame graph we get the following:

->  aws.lambda foo.service
            --> Initialization
                               --> aws.lambda.foobar foo.service (error)

We are using the Datadog log forwarder and the below options set in the SAM config.

  FooFunc:
    Type: AWS::Serverless::Function
    Properties:
         ...
      Environment:
        Variables:
          DD_TRACE_ENABLED: true
          DD_FLUSH_TO_LOG: true
      Tags:
        env: dev
        service: "aws.lambda.foo"
        version: v1.0.0

Specifications

Datadog Go Lambda package version: v0.9.0
Datadog Tracer package version: v1.28.0
Go version: 1.15

Document the AWS Layer ARN for each release in the same way that the Node repo does

Expected Behavior

Node Lambda Layer Release Tag

Actual Behavior

Golang Lambda Layer Release Tag

Steps to Reproduce the Problem

I cannot figure out what ARN I should use for this repo.

Per Datadog's documentation, the format is

arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-<RUNTIME>:<VERSION>

However, per the documentation here: https://docs.datadoghq.com/integrations/amazon_lambda/?tab=go#installing-and-using-the-datadog-layer, Go's RUNTIME value is: N/A.

Given that, for the life of me, I cannot figure out the format of a valid ARN for this repo.

Adding an example to either the Readme notes or each release tag (a la the Node repo) would be a tremendous help.

How can I add connections to an SFTP server to the Service Map?

My GoLang AWS Lambda connects to an SFTP server and uploads a file to it.
It uses golang.org/x/crypto/ssh and github.com/pkg/sftp to do so.

How should I instrument the Go code to have these connections on the APM Service Map?

Not running handler

I have the following code, which I can see is building and connects to Datadog but it seems to not execute anything in the myHandler function.
I don't see the log entries appear in CloudWatch or the metric appear in DD.

Any ideas where I have gone wrong?

Goland is complaining that build constraints exclude all GO files, but it builds ok. I set go.mod to 1.18

package main

import (
	"context"
	"fmt"
	ddLambda "github.com/DataDog/datadog-lambda-go"
	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-lambda-go/lambda"
	log "github.com/sirupsen/logrus"
	"os"
)

func init() {
	log.SetFormatter(&log.JSONFormatter{
		PrettyPrint: false,
	})
}

func main() {
	// Start the lambda handler
	lambda.Start(ddLambda.WrapHandler(myHandler, nil))
}

func myHandler(ctx context.Context, event events.KafkaEvent) (string, error) {
	log.Info(fmt.Sprintf("Running Lambda against region: %s", getEnv("AWS_REGION", "eu-west-1")))
	lenEventRecords := len(event.Records)
	if lenEventRecords > 0 {
		log.Info(fmt.Sprintf("Kafka records recieved: %+v", lenEventRecords))
		for key, element := range event.Records {
			log.Info(fmt.Sprintf("%+v, %+v", key, element))
		}
		ddLambda.Metric(
			"redpanda.event_received",           // Metric name
			float64(lenEventRecords),            // Metric value
			"topic:test", "producer:producer_1", // Associated tags
		)
	}
	return "", nil
}

// GetEnv Simple helper function to read an environment or return a default value
func getEnv(key string, defaultVal string) string {
	if value, exists := os.LookupEnv(key); exists {
		return value
	}
	return defaultVal
}

Filter logs before forwarding to datadog

Hello,

if I understand correctly, this lambda listens to cloudwatch logs, and forwards them to datadog.

Is it possible to filter out unwanted logs, for example that contain a message that matches a list of different patterns, so that these are not sent to datadog?

Add support for tracer Env and Version tags

I reported first this bug in DataDog/dd-trace-go but I was redirected here. I have problem to set proper Env and Version tag for my lambda -> redis connection.

Code example

import (
	"context"
	"encoding/json"
	"errors"

	ddlambda "github.com/DataDog/datadog-lambda-go"
	golambda "github.com/aws/aws-lambda-go/lambda"
)

func (d *deps) lambdaHandler(ctx context.Context) error {
...

type deps struct {
	ElastiCache      redis.UniversalClient
}

func main() {
	d := deps{
		ElastiCache: redistrace.NewClient(
			&redis.Options{
				Addr: os.Getenv("elastiCacheUrl"),
			},
			redistrace.WithServiceName("ElastiCache"),
			redistrace.WithAnalytics(true),
		),
	}

       config = &ddlambda.Config{
		DDTraceEnabled:        true,
		MergeXrayTraces:       true,
		ShouldUseLogForwarder: true,
	}

        golambda.Start(ddlambda.WrapHandler(d.lambdaHandler, config))
}

LInk to dd-trace context issue DataDog/dd-trace-go#1102

Support for monitoring invocation payloads

Expected Behavior

Due to this blog post - https://www.datadoghq.com/blog/troubleshoot-lambda-function-request-response-payloads/ and this documentation https://docs.datadoghq.com/serverless/distributed_tracing/collect_lambda_payloads/ we should be able to monitor invocation payloads. From what is presented in the second link Go runtime is not supported yet.

Handle panic in handle code?

Expected Behavior

I am sorry I am not sure if it's a bug report, something I misunderstood or...

Basically, I noticed that since I migrated my lambda to this library to leverage APM/logs/metrics in Datadog (which is pretty awesome), I am not able to monitor panics anymore. I have the following code:

func main() {
	lambda.Start(ddlambda.WrapFunction(handle, nil))
}

and the handle method returns (string, error) as expected. But if the code panics instead of returning an error, then I do not see any trace in APM and I have an INFO log that is not correctly parsed that looks like this:

{"errorMessage":"interface conversion: interface {} is nil, not string","errorType":"TypeAssertionError","stackTrace":[{"path":"github.com/aws/[email protected]/lambda/errors.go","line":39,"label":"lambdaPanicResponse"},{"path":"github.com/aws/[email protected]/lambda/invoke_loop.go","line":122,"label":"callBytesHandlerFunc.func1"},{"path":"runtime/panic.go","line":914,"label":"gopanic"},{"path":"runtime/iface.go","line":263,"label":"panicdottypeE"},{"path":"work/warpstream/warpstream/scripts/lambdas/signups/signups.go","line":114,"label":"CheckRecentSignups"},{"path":"work/warpstream/warpstream/scripts/lambdas/signups/signups.go","line":62,"label":"RunLamdba"},{"path":"common/main.go","line":71,"label":"handle"},{"path":"reflect/value.go","line":596,"label":"Value.call"},{"path":"reflect/value.go","line":380,"label":"Value.Call"},{"path":"github.com/!data!dog/[email protected]/internal/wrapper/wrap_handler.go","line":161,"label":"callHandler"},{"path":"github.com/!data!dog/[email protected]/internal/wrapper/wrap_handler.go","line":61,"label":"WrapHandlerWithListeners.func1"},{"path":"reflect/value.go","line":596,"label":"Value.call"},{"path":"reflect/value.go","line":380,"label":"Value.Call"},{"path":"github.com/aws/[email protected]/lambda/handler.go","line":355,"label":"reflectHandler.func2"},{"path":"github.com/aws/[email protected]/lambda/invoke_loop.go","line":125,"label":"callBytesHandlerFunc"},{"path":"github.com/aws/[email protected]/lambda/invoke_loop.go","line":75,"label":"handleInvoke"},{"path":"github.com/aws/[email protected]/lambda/invoke_loop.go","line":39,"label":"startRuntimeAPILoop"},{"path":"github.com/aws/[email protected]/lambda/entry.go","line":106,"label":"start"},{"path":"github.com/aws/[email protected]/lambda/entry.go","line":69,"label":"StartWithOptions"},{"path":"github.com/aws/[email protected]/lambda/entry.go","line":45,"label":"Start"},{"path":"common/main.go","line":90,"label":"main"},{"path":"runtime/proc.go","line":267,"label":"main"},{"path":"runtime/asm_amd64.s","line":1650,"label":"goexit"}]}

Actual Behavior

I was hoping to get a failing trace in APM and an error log attached to it.

Specifications

Datadog Go Lambda package version: v1.13.0
Go version: 1.22

Support fire and forget?

We are currently facing a problem when sending metrics from lambda from some of our AWS regions: for some of them the lambda execution time almost duplicate if we send metrics through the client; an option we have been evaluating is switching to Cloudwatch logs + forwarder, but that implies having to create infra, some delay, extra costs, etc.

Whereas metrics are important to us, loosing some data from time to time isn't that concerning, wonder if there is plan / possibility to support sending the metrics in a "fire and forget" way so we don't have to wait for server ACK and we can simply stop the lambda after sending the metrics.

release with fasthttp lib 1.35

I would like to know when will be released a version with the version 1.35 of fasthttp lib, I am receiving some security alerts because that.

Update github.com/aws/aws-xray-sdk-go to latest version

The current version (1.6) of aws-xray-sdk-go of https://github.com/aws/aws-xray-sdk-go/blob/v1.6.0/go.mod#L15 has outdated dependencies (fasthttp 1.24.0 ) with vulnerabilities.
More Details https://security.snyk.io/vuln/SNYK-GOLANG-GITHUBCOMVALYALAFASTHTTP-2407866
updating aws-xray-sdk-go to version >= 1.7 should fix it.

Support for DD_API_KEY_SECRET_ARN environment variable

Expected Behavior

Datadog agent for serverless (https://github.com/DataDog/datadog-agent/blob/main/cmd/serverless/README.md#serverless-environment-configuration) supports retrieving datadog API key via secret, but this library didn't support it. Currently this library supports datadog API key via DD_API_KEY or DD_KMS_API_KEY environment variables. It would be really helpful for this library to support datadog API key via secret too so my AWS Lambda function can provide 1 environment variable only (DD_API_KEY_SECRET_ARN) for both the library and datadog agent extension.

Enhanced metrics always flushed to logs

Expected Behavior

According to the documentation for the Datadog Lambda Extension, enhanced metrics should be sent to Datadog by the extension when it is enabled, rather than going through CloudWatch logs.

Actual Behavior

Enhanced metrics are being force pushed to logs while custom metrics are not:

datadog-lambda-go/internal/metrics/listener.go

Line 202 in 757bd4f

    
           l.AddDistributionMetric(fmt.Sprintf("aws.lambda.enhanced.%s", metricName), 1, time.Now(), true, tags...)

We are able to receive all custom metrics in our dashboards but no enhanced metrics are showing up.

An example from a lambda function running with the Datadog Lambda Extension and DD_FLUSH_TO_LOG = false

// enhanced metrics being flushed to logs and do not show up
2021/09/29 23:20:58 {"status":"debug","message":"datadog: sending metric via log forwarder"}
{"m":"aws.lambda.enhanced.invocations","v":1,"e":1632957658,"t":["functionname:***","region:eu-west-1","account_id:***","memorysize:128","cold_start:false","datadog_lambda:v1.3.0","resource:***","dd_lambda_layer:datadog-go1.15.15"]}

// custom metrics are not flushed to logs 
2021/09/29 23:20:58 {"status":"debug","message":"datadog: adding metric \"***\", with value 1.000000"}

Is this a bug in the code or are we still expected to run a Datadog Forwarder along side the Datadog Lambda extension in order to get enhanced metrics?

Steps to Reproduce the Problem

Run a lambda function with DataDog Lambda Extension and enhanced metrics enabled
Check that only non enhanced metrics are showing up in datadog

Specifications

Datadog Go Lambda package version: v1.3.0
Go version: 1.15

DD Lambda Extension - connection refused (even though the extension layer is added)

Expected Behavior

We are trying to set up distributed tracing, where a Golang lambda is the entrypoint of our flow that we want to trace. We want to use the lambda extension, and are using the datadog-lambda-go package pointing to the maxday/support-tracing-with-extension branch of @maxday.

The expected behaviour would be that the package notices that the extension is installed, and that it can do a '/hello' call, so that afterwards the dd-lambda-go package can send traces towards the agent, instead of printing them to stdout for the log forwarder to use.

Actual Behavior

We receive the following debug message:
"Will use the API since the Serverless Agent was detected but the hello route was unreachable".

This means that the package correctly detects that the extension is present, but that it doesn't get a 200 OK response from the /hello endpoint of the extension. When we do some debugging ourselves in the lambda, we get a connection refused when trying to query http://localhost:8124/lambda/hello, which explains the debug message.

We were wondering, since @maxday seemed to get it working, if there was something specific for the Golang runtime we need to do, to get this setup working. Based on the documentation of extensions, it should automatically run as a separate process, so not sure what we are doing wrong.

Thanks in advance for any insights!

Steps to Reproduce the Problem

Install the Datadog Lambda extension as a layer (arn:aws:lambda:eu-central-1:464622532012:layer:Datadog-Extension:9) => we tried both 8 and 9, same issue.
Use the "datadog-lambda-go@maxday/support-tracing-with-extension" package to send traces
The package fails to connect to the extension agent

Specifications

Datadog Go Lambda package version: datadog-lambda-go@maxday/support-tracing-with-extension
Go version: go1.15.14

Stacktrace

{
  "date": "2021-07-16T10:14:44Z",
  "os_name": "Linux (Unknown Distribution)",
  "os_version": "2018.03",
  "version": "v1.31.1",
  "lang": "Go",
  "lang_version": "go1.15.14",
  "env": "development",
  "service": "aws.lambda",
  "agent_url": "http://localhost:8126/v0.4/traces",
  "agent_error": "",
  "debug": false,
  "analytics_enabled": false,
  "sample_rate": "NaN",
  "sampling_rules": null,
  "sampling_rules_error": "",
  "tags": {
      "_dd.origin": "lambda",
      "runtime-id": "6474742f-c437-4545-bf05-873c06ba50c7"
  },
  "runtime_metrics_enabled": false,
  "health_metrics_enabled": false,
  "dd_version": "",
  "architecture": "amd64",
  "global_service": "aws.lambda",
  "lambda_mode": "true",
  "agent_features": {
      "DropP0s": false,
      "V05": false,
      "Stats": false
  }
}

Support Opentracing bridge

dd-tracer has support for the opentracer, however lambdas can't directly use Opentracing.

Unable to add tags to Span created by Lambda Wrapper

Expected Behavior

After using tracer.SpanFromContext() to get the span, the function span.SetTag() should set a tag on the span.

Actual Behavior

No tags are added, and none can be changed.

Steps to Reproduce the Problem

Wrap your lambda handler with ddlambda.WrapFunction
Get the span from the context in your handler with tracer.SpanFromContext()
Use span.SetTag() to set a custom tag
Check APM. The tag is not present on the span.

Specifications

Datadog Go Lambda package version: 1.8.0
Go version: 1.19.x

Stacktrace

N/A

upgrade dd-trace-go to v1.41.0 or newer

Expected Behavior

Running on Go v 1.19 is supported without errors.

Actual Behavior

I found the program got an error about gc and it exited with a panic.
It seems to be related to dependent module dd-trace-go.

https://github.com/DataDog/datadog-lambda-go/blob/main/go.mod#L13

Please refer this resolved issue; DataDog/dd-trace-go#1455

Specifications

Datadog Go Lambda package version: 1.6.0
Go version: 1.19

Stacktrace

2022-09-09T15:31:31.249+09:00   panic: Something in this program imports go4.org/unsafe/assume-no-moving-gc to declare that it assumes a non-moving garbage collector, but your version of go4.org/unsafe/assume-no-moving-gc hasn't been updated to assert that it's safe against the go1.19 runtime. If you want to risk it, run with environment variable ASSUME_NO_MOVING_GC_UNSAFE_RISK_IT_WITH=go1.19 set. Notably, if go1.19 adds a moving garbage collector, this program is unsafe to use.
panic: Something in this program imports go4.org/unsafe/assume-no-moving-gc to declare that it assumes a non-moving garbage collector, but your version of go4.org/unsafe/assume-no-moving-gc hasn't been updated to assert that it's safe against the go1.19 runtime. If you want to risk it, run with environment variable ASSUME_NO_MOVING_GC_UNSAFE_RISK_IT_WITH=go1.19 set. Notably, if go1.19 adds a moving garbage collector, this program is unsafe to use.

2022-09-09T15:31:31.249+09:00	goroutine 1 [running]:

2022-09-09T15:31:31.249+09:00	go4.org/unsafe/assume-no-moving-gc.init.0()

2022-09-09T15:31:31.249+09:00	/go/pkg/mod/go4.org/unsafe/[email protected]/untested.go:25 +0x1f4

2022-09-09T15:31:31.250+09:00	EXTENSION Name: datadog-agent State: Registered Events: [INVOKE,SHUTDOWN]

2022-09-09T15:31:31.285+09:00	LOGS Name: datadog-agent State: Already subscribed Types: [platform,function,extension]

Dry run options?

Expected Behavior

Usually, when developing a lambda in Go, I prefer to doing something like:

package main

var (
   DryRun = true
)

func init() {
  //...
  if lambdaFunctionName := os.Getenv("AWS_LAMBDA_FUNCTION_NAME"); lambdaFunctionName != "" {
    DryRun = false
  }
}

func main() {
  if DryRun {
    // invoke manually
    // ...
    ctx := context.Background()
    response, err := handler(ctx)
    // ...
  } else {
    lambda.Start(handler)
  }
}

func handler(ctx context.Context) (string, error) {
   // ... 
}

But I'm having a hard time figuring out how to do this w/ ddlambda in the mix.

I got as far as doing:

		wrapped := ddlambda.WrapHandler(handler, nil)
		h, ok := wrapped.(func(ctx context.Context, msg json.RawMessage) (interface{}, error))
		if !ok {
			log.Fatalf("[FATAL] [DRYRUN] Could not unwrapped ddlambda")
		}

		h(ctx, json.RawMessage("{}"))

Which seems to work, but seems a bit less than optimal, and I haven't actually seen my custom Metrics show up in DataDog yet.

Actual Behavior

Is there a better, more supported way to run the lambda locally while still testing DD metrics, etc.?

Specifications

Datadog Go Lambda package version: 0.3.0
Go version: 1.13.1

Status Code 413, Body Payload too big

Expected Behavior

Should be able to send payload. Some of the solutions for other DataDog libraries suggest to compress payload.

Actual Behavior

{
    "status": "error",
    "message": "datadog: failed to flush metrics to datadog API: Failed to send metrics to API. Status Code 413, Body Payload too big"
}

Specifications

Go version: 1.14.2

Stacktrace

{
  "status": "error",
  "message": "datadog: failed to flush metrics to datadog API: Failed to send metrics to API. Status Code 413, Body Payload too big"
}

Metric using flush to log not seeing in datadog dashboard

Expected Behavior

DD_FLUSH_TO_LOG: true, It should be able to send metric to datadog using log forwarder.
DD_FLUSH_TO_LOG: false, It should be able to send metric to datadog without using log forwarder.

Actual Behavior

DD_FLUSH_TO_LOG: true, It should be able to send metric to datadog using log forwarder. Even though metric can be found in cloudwatch log, it is not sending to datadog dashboard.

Steps to Reproduce the Problem

DD_FLUSH_TO_LOG = true
See cloudwatch logs, it can be found.
The custom metric can't be found in dashboard

Specifications

Datadog Go Lambda package version: v0.9.0
Go version: go1x

Stacktrace

{
  "m": "vss.catalog.success",
  "v": 1,
  "e": 1605693166,
  "t": [
      "dh_squad:vss",
      "dh_tribe:new business",
      "dh_env:dev",
      "app:agent_service",
      "v:1",
      "dd_lambda_layer:datadog-go1.14.3"
  ]
}

Traces not seeing in datadog dashboard

Expected Behavior

Trace should be able to see in datadog dashboard.

Actual Behavior

Although the traces can be able to see it in cloudwatch logs, it is not available in datadog dashboard ui

Steps to Reproduce the Problem

Send trace using library, DD_TRACE_ENABLED = true
Inspect the logs in cloudwatch
Only trace generated from aws xray is visible in datadog dashboard

Specifications

Datadog Go Lambda package version: v0.9.0
Go version: go1.x

Stacktrace

{
              "trace_id": "7edf85708431adc6",
              "span_id": "7c97152fd75df086",
              "parent_id": "7edf85708431adc6",
              "name": "child.span",
              "resource": "child.span",
              "error": 0,
              "meta": {
                  "__dd.origin": "lambda",
                  "runtime-id": "bee3decd-617d-4c42-bfe7-ef51db1fb58d",
                  "env": "dev"
              },
              "metrics": {
                  "_sampling_priority_v1": 1
              },
              "start": 1605684489077109414,
              "duration": 100055542,
              "service": "aws.lambda"
          }

Error values are logged as INFO level logs instead of ERRORs in datadog

Expected Behavior

Expected the error to be have a level:error field on the log message so Datadog correct intrepets the log.

Actual Behavior

The Datadog Log Forwarder function doesn't seem to add an level:error to log messages that are actually errors that have bubbled up to the final lambda handler.

So it's displayed as an INFO log in Datadog instead of ERROR log

Steps to Reproduce the Problem

Have a simple handler like this with an error:

func handler(ctx context.Context, sqsEvent events.SQSEvent) error {

	err := processSQSEvent(ctx, &sqsEvent)
	if err != nil {
		return errors.Wrap(err, "error processing the SQS event")
	}
	return nil
}

func main() {
	lambda.Start(ddlambda.WrapHandler(handler, nil))
}

If processSQSEvent returns an err you will see "error processing the SQS event" without a log level field being forwarded to datadog.

How can I resolve this?

Do I need to format the error as a JSON log when returning it?

Specifications

Datadog Go Lambda package version: github.com/DataDog/datadog-lambda-go v0.7.0
Go version: 1.16

Latest version is unavailable

Expected Behavior

Run go mod vendor
add github.com/DataDog/datadog-lambda-go v0.5.0 to go.mod file

Actual Behavior

github.com/DataDog/datadog-lambda-go v0.2.5 added to go.mod file

Steps to Reproduce the Problem

sam init
select go template
run go mod init and setup package name
add "github.com/DataDog/datadog-lambda-go"
run go mod vendor

Specifications

Go version: 1.13

Custom metrics with percentile calculations

Expected Behavior

I would like to be able to submit a custom metric from the lambda extension, and then be able to analyze that metric's percentile calculations once the metrics have arrived in DataDog. This would more closely match with my experience submitting Distribution metrics via DogStatsD.

Actual Behavior

I can only aggregate the metrics I generated through the Lambda extension by avg, min, max, sum, and count.

Steps to Reproduce the Problem

Send any custom metric from a Lambda function.
Percentile aggregations are not available:

Specifications

Datadog Go Lambda package version: 43
Go version: provided.al2

Flush Metrics

I am using the DD Metrics function to publish metrics scraped from a Prometheus endpoint over to Datadog via a lambda.

Is there a way to flush the metrics into batches as in each scrape I am getting around 35,000 metrics which If I just call

		ddLambda.Metric(
			v.Name,  // Metric name
			v.Value, // Metric value
			tags..., // Associated tags
		)

Then I get the error -
"datadog: failed to flush metrics to datadog API: with no retry: Failed to send metrics to API. Status Code 413
Along with Payload to large.

Thanks :)

Support for SetUser

dd-trace-go released version 1.37.0 (https://github.com/DataDog/dd-trace-go/releases/tag/v1.37.0) with new feature - tracer.SetUser - I have a question/feature request. Is there a possibility for a lambda using datadog-lambda-go to set dynamically (not by lambda environment variables) during lambda execution User details that will help in better tracing/debuging?

Tracing debugging is always turned on

Expected Behavior

Datadog trace debugging is controlled by explicit code configuration or environment variables.

Actual Behavior

The Datadog trace library is always initialised with the debug mode on, and this cannot be switched off.

This is related to this line:

datadog-lambda-go/internal/trace/listener.go

Line 64 in 42c4a21

tracer.WithDebugMode(true),

The tracer is configured with tracer.WithDebugMode(true), forcing debug on.

Steps to Reproduce the Problem

Enable the library on a lambda with tracing enabled as per the instructions

Specifications

Datadog Go Lambda package version: 1.0.0
Go version: 1.16.x

Inherit tags from Lambda function

Expected Behavior

Metrics submitted with the library have the same tags as the Lambda function, plus the tags the user provides programmatically.
In case one tag is present in the Lambda function and at the same time is set programmatically the latter remains.

Actual Behavior

Only tags provided programmatically are submitted in metrics.

Steps to Reproduce the Problem

Create a Lambda function that submits a distribution metric using the library
Add some tags to the Lambda function in AWS
Execute the function
The tags of the Lambda function are not present in the metric summary.

Specifications

Datadog Go Lambda package version: 0.4.0
Go version: 1.13.1

Stacktrace

Does not apply

xray trace_id should be uint64 not hex

I already described this in - #44

I would like to ask, are you sure that x-ray mode is working with Datadog traces? From tests, I see that it is generating trace_id as hex

{
    "traces": [
        [
            {
                "trace_id": "6fb744bf13869d00",
                "span_id": "1e8538262a43c188",
                "parent_id": "1e5df70157538a8e",
                "name": "aws.lambda",

And traces are not connecting with logs - https://docs.datadoghq.com/tracing/connect_logs_and_traces/ - because Datadog is making query like @lambda.request_id:(fe757c85-ef6f-4c09-8e24-ca5a16edbdae) OR trace_id:8049978446405082368 (converted hex to uint64 is giving correct value). Also reading documentation https://docs.datadoghq.com/api/v1/tracing/#send-traces looks like DataDog is expecting trace_id as uint64

Even in #44 we can find

fmt.Println("Current trace ID: " + strconv.FormatUint(currentSpanContext.TraceID(), 10))

Bottom line: In current Hex format traces are not connected to Logs - it should be fixed and send as uint64

What do you think? Or maybe I am doing something wrong

CC @nhinsch @DarcyRaynerDD

Support Passing in Timestamp

Hey all,

I'm looking into moving a particular Lambda from logging MONITORING lines to using the Lambda Layer + libraries for metric consistency purposes (the crawler has some unexpected behavior which causes some metrics to be dropped). However, we have a use case where we'd like to be able to specify the timestamp used when creating metrics, which currently isn't possible.

I'm not sure if adding Timestamp support is something you'd want to support or if its too niche, but what are y'alls thoughts? We could possibly add a new function:

func DistributionWithTimestamp(metric string, value float64, timestamp int64, tags ...string)

After upgrade from v1.4.0 to v1.5.0 got many vulnerabilities

pkg:golang/github.com/go-ldap/ldap/[email protected]
1 known vulnerabilities affecting installed version 
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 1 vulnerability found                                                                            ┃
┣━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Description        ┃ 1 non-CVE vulnerability found. To see more details, please create a free    ┃
┃                    ┃ account at https://ossindex.sonatype.org/ and request for this information  ┃
┃                    ┃ using your registered account                                               ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ OSS Index ID       ┃ sonatype-2020-1055                                                          ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ CVSS Score         ┃ 8.1/10 (High)                                                               ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ CVSS Vector        ┃ CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:N                                ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Link for more info ┃ https://ossindex.sonatype.org/vulnerability/sonatype-2020-1055              ┃
┗━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
pkg:golang/github.com/gofiber/fiber/[email protected]
1 known vulnerabilities affecting installed version 
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 1 vulnerability found                                                                            ┃
┣━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Description        ┃ 1 non-CVE vulnerability found. To see more details, please create a free    ┃
┃                    ┃ account at https://ossindex.sonatype.org/ and request for this information  ┃
┃                    ┃ using your registered account                                               ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ OSS Index ID       ┃ sonatype-2021-4750                                                          ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ CVSS Score         ┃ 6.5/10 (Medium)                                                             ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ CVSS Vector        ┃ CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H                                ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Link for more info ┃ https://ossindex.sonatype.org/vulnerability/sonatype-2021-4750              ┃
┗━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
pkg:golang/github.com/hashicorp/vault/[email protected]
1 known vulnerabilities affecting installed version 
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 1 vulnerability found                                                                            ┃
┣━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Description        ┃ 1 non-CVE vulnerability found. To see more details, please create a free    ┃
┃                    ┃ account at https://ossindex.sonatype.org/ and request for this information  ┃
┃                    ┃ using your registered account                                               ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ OSS Index ID       ┃ sonatype-2021-3619                                                          ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ CVSS Score         ┃ 7.5/10 (High)                                                               ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ CVSS Vector        ┃ CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H                                ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Link for more info ┃ https://ossindex.sonatype.org/vulnerability/sonatype-2021-3619              ┃
┗━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
pkg:golang/github.com/jinzhu/[email protected]
1 known vulnerabilities affecting installed version 
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ [CVE-2019-15562] CWE-89: Improper Neutralization of Special Elements used in an SQL Command ('SQL Injection')                                                                                                     ┃
┣━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Description        ┃ ** DISPUTED ** GORM before 1.9.10 allows SQL injection via incomplete                                                                                                                        ┃
┃                    ┃ parentheses. NOTE: Misusing Gorm by passing untrusted user input where Gorm                                                                                                                  ┃
┃                    ┃ expects trusted SQL fragments is a vulnerability in the application, not in                                                                                                                  ┃
┃                    ┃ Gorm.                                                                                                                                                                                        ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ OSS Index ID       ┃ CVE-2019-15562                                                                                                                                                                               ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ CVSS Score         ┃ 9.8/10 (Critical)                                                                                                                                                                            ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ CVSS Vector        ┃ CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H                                                                                                                                                 ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Link for more info ┃ https://ossindex.sonatype.org/vulnerability/CVE-2019-15562?component-type=golang&component-name=github.com%2Fjinzhu%2Fgorm&utm_source=nancy-client&utm_medium=integration&utm_content=1.0.37 ┃
┗━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
pkg:golang/github.com/urfave/[email protected]
1 known vulnerabilities affecting installed version 
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ 1 vulnerability found                                                                            ┃
┣━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Description        ┃ 1 non-CVE vulnerability found. To see more details, please create a free    ┃
┃                    ┃ account at https://ossindex.sonatype.org/ and request for this information  ┃
┃                    ┃ using your registered account                                               ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ OSS Index ID       ┃ sonatype-2021-1485                                                          ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ CVSS Score         ┃ 6.1/10 (Medium)                                                             ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ CVSS Vector        ┃ CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:C/C:L/I:L/A:N                                ┃
┣━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ Link for more info ┃ https://ossindex.sonatype.org/vulnerability/sonatype-2021-1485              ┃
┗━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Please update dependencies

latest version could not be accepted by go mod

Expected Behavior

Run go mod vendor
add github.com/DataDog/datadog-lambda-go v0.5.0 to go.mod file

Actual Behavior

github.com/DataDog/datadog-lambda-go v0.2.5 added to go.mod file

latest version "0.5.0" is not semver tag, and it should be "v0.5.0"

Steps to Reproduce the Problem

sam init
select go template
run go mod init and setup package name
add "github.com/DataDog/datadog-lambda-go"
run go mod vendor

Specifications

Go version: 1.13

Include the layer arn's on the releases like all the other DataDog lambda layer repos

Expected Behavior

On the releases for this repo, the set of arn's for the layer versions this release correspond to is included in a code block in the release description.

Examples:

Java: arn:aws:lambda:<AWS_REGION>:464622532012:layer:dd-trace-java:8
Ruby: arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Ruby2-7:16
JS: arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Node12-x:88
Python: arn:aws:lambda:<AWS_REGION>:464622532012:layer:Datadog-Python37:69
Go: arn:aws:lambda:<AWS_REGION>:464622532012:layer:????:??

Actual Behavior

There is no trace whatsoever anywhere in the repo or DataDog docs about what layer arns/versions are, or even what their format looks like for golang.

Working at a company that restricts what's possible to do in the AWS console means we cannot just browse for simple things like this or even take wild guesses. It's inconsistent with all of the other datadog lambda layer repositories. This would be a small change to release behavior, but make a huge impact for end users (in my opinion, or at least for other devs finding themselves in my unfortunate situation).

If anything, please consider this for the sake of consistency and bc it will improve the experience of your end users, even if just a little. <3

Steps to Reproduce the Problem

Go to literally any other DataDog/datadog-lambda-* releases page and look for arn
See the releases in an easy to read code block
Got to this repo's releases page, look for arn to no avail.
Suffer X_X

Specifications

Datadog Go Lambda package version: Literally all of them.
Go version: N/A

Stacktrace

N/A

Cross-service tracing breaks when `x-datadog-sampling-priority` header is not present

Expected Behavior

The x-datadog-sampling-priority header should (?) be optional; cross-service traces and X-ray merging should still work when this header is missing.

The ddtrace library header injector treats the field as optional, so it may not always be present:
https://github.com/DataDog/dd-trace-go/blob/v1/ddtrace/tracer/textmap.go#L225-L227

I came across this behaviour when there was an intermediate service whose runtime doesn't have a first-party library. The traceID and parentID headers were forwarded, seemingly satisfying the requirements of the ddtrace-go library's extractor, but this library implements its own behaviour as part of support for Xray trace merging.

Actual Behavior

If the x-datadog-trace-id and x-datadog-parent-id headers are present, but the x-datadog-sampling-priority header is missing, the trace context is not propagated and the traces aren't joined in the Datadog interface.

Specifications

Datadog Go Lambda package version: 1.0.0
Go version: 1.16.x

Thread Safety, Safe For Sharing Between Multiple Goroutines

Sorry, i don't think this was correct, will do some more research, sorry about this.

Unable to set the parent span for new child spans

I'm using the below code in an attempt to trace individual SQS messages and add them to the parent span the other service that sent them.

I have confirmed that metadata contains both x-datadog-parent-id and x-datadog-trace-id in the format that the tracer.TextMapCarrier requires.

The traces are coming through ok but the issue is that by the time the traces are sent through the Datadog Lambda Forwarder the parent is set to the span started for the root of the Lambda execution (in listener.go#startFunctionExecutionSpan) instead of the that has been extracted.

opts := []ddtrace.StartSpanOption{
	tracer.ResourceName("eventHandler"),
}
if parentTraceCtx, err := tracer.Extract(tracer.TextMapCarrier(metadata)); err == nil {
	opts = append(opts, tracer.ChildOf(parentTraceCtx))
}

tracer.StartSpanFromContext(ctx, "Process", opts...)

Is there a limitation that I am perhaps misunderstanding when trying to set the parent span to something other than the Lambda execution span?

Steps to Reproduce the Problem

See above code snippet.

Specifications

Datadog Go Lambda package version: v1.0.1
Go version: v1.16
X-Ray tracing: INACTIVE

Stacktrace

N/A

enable DD_TAGS env var

The Datadog documentation frequently make reference to an env var DD_TAGS for articulating a list of tags to include in events emitted to Datadog -- some examples:

This enabling this would make for a really great developer experience.

Expected Behavior

running a program with this env var:

DD_TAGS=foo:bar main

with this code (as per readme)

ddlambda.Metric(
  "coffee_house.order_value", // Metric name
  12.45, // The value
  "product:latte", "order:online" // Associated tags
)

should yield metrics in datadog with the following tags:

foo:bar
product:latte
order:online

Support for extraction of B3 headers

Would like to have support for extracing incomming B3 headers.
Current solution only parses xray and datadog trace id and span id.

Suggestion is to use the implementation found in dd-trace-go/tracer using env variables DD_PROPAGATION_STYLE_EXTRACT

Version tag not appearing in traces

Expected Behavior

We set the DD_VERSION environment variable, and the version tag on the lambda, to get version tagging in our lambda traces. DD_SERVICE is used to set the application name. Expected behavior is for the traces to contain the version tag.

We also use the log forwarder, and set the DD_FLUSH_TO_LOG environment variable.

Actual Behavior

version tag not present in traces.

It seems the internal tracer started by datadog-lambda-go sets the service name to aws-lambda, which, when setting DD_SERVICE to our application name, make this in the tracer implementation exclude the version tag.

The service tag in the traces we get is set to the value of the DD_SERVICE environment variable.

Steps to Reproduce the Problem

Set DD_VERSION and DD_SERVICE
version missing in traces

Specifications

Datadog Go Lambda package version: v0.9.0
Datadog Tracer package version: v1.27.1
Go version: 1.15

Stacktrace

upgrade dd-trace-go to v1.60.2 or newer

Expected Behavior

Be able to download datadog-lambda-go without inet.af/netaddr errors

Actual Behavior

With my golang cache is cleaned, I'm trying to download the datadog-lambda-go package and then I'm getting an error to download the inet.af/netaddr dependency (indirect dependency, more details below).

Looks like the dd-trace-go fixed that issue on version v1.60.2, so it should be nice to bump it in here.

Here is dd-trace-go PR for reference: DataDog/dd-trace-go#2553

Steps to Reproduce the Problem

Clean the go cache: go clean -cache && go clean -modcache
Try to download the datadog-lambda-go: go get github.com/DataDog/datadog-lambda-go
You should see the error: inet.af/netaddr: unrecognized import path "inet.af/netaddr"

Specifications

Datadog Go Lambda package version: v1.15.0
Go version: 1.20

Stacktrace

github.com/DataDog/datadog-lambda-go imports
github.com/DataDog/datadog-lambda-go/internal/trace imports
gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer imports
gopkg.in/DataDog/dd-trace-go.v1/internal/appsec imports
gopkg.in/DataDog/dd-trace-go.v1/internal/appsec/listener/grpcsec imports
gopkg.in/DataDog/dd-trace-go.v1/internal/appsec/emitter/grpcsec imports
github.com/DataDog/appsec-internal-go/netip imports
inet.af/netaddr: unrecognized import path "inet.af/netaddr": https fetch: Get "https://inet.af/netaddr?go-get=1": dial tcp: lookup inet.af on 172.31.0.2:53: no such host

Build time increased 8x from including package

Expected Behavior

I expected this dep to add a minor amount of build time to my Lambda.

Comment

Lambdas are usually small, simple pieces of logic, so I was surprised when adding a single dependency caused our build time to increase so dramatically. I believe this is related to the number dependencies this package has.

Actual Behavior

My build time has increased by 150+ seconds.

Before:

[+] Building 94.5s (18/18) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                     0.1s
 => => transferring dockerfile: 37B                                                                                                                                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                                                          0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                                                                                               1.6s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:443aab4ca21183e069e7d8b2dc68006594f40bddf1b15bbd83f5137bd93e80e2                                                                                                          0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                        0.0s
 => [internal] load metadata for public.ecr.aws/lambda/provided:al2                                                                                                                                                                      0.5s
 => CACHED [build 1/8] FROM public.ecr.aws/lambda/provided:al2@sha256:33ab84899549495234fd55839a00e1baf9406e80bc16fd12b77f97729283c051                                                                                                   0.0s
 => [internal] load build context                                                                                                                                                                                                        0.1s
 => => transferring context: 164B                                                                                                                                                                                                        0.0s
 => CACHED [build 2/8] RUN yum install -y golang                                                                                                                                                                                         0.0s
 => CACHED [build 3/8] RUN go env -w GOPROXY=direct                                                                                                                                                                                      0.0s
 => CACHED [build 4/8] WORKDIR /app                                                                                                                                                                                                      0.0s
 => CACHED [build 5/8] COPY go.mod go.sum ./                                                                                                                                                                                             0.0s
 => [build 6/8] RUN go mod download                                                                                                                                                                                                     0.0s
 => [build 7/8] COPY *.go ./                                                                                                                                                                                                             0.3s
 => [build 8/8] RUN go build -o /app/main                                                                                                                                                                                               20.5s  <---
 => [stage-1 2/2] COPY --from=build /app/main /main                                                                                                                                                                                      0.3s
 => exporting to image                                                                                                                                                                                                                   0.6s
 => => exporting layers                                                                                                                                                                                                                  0.5s
 => => writing image sha256:7289d70543610a88d7283e11a7dff75884be310819ac6f9dfda92da4e8907ecf                                                                                                                                             0.0s

After:

[+] Building 194.0s (19/19) FINISHED
 => [internal] load build definition from Dockerfile                                                             0.3s
 => => transferring dockerfile: 37B                                                                              0.1s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 2B                                                                                  0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                       2.2s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                                                 0.0s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:443aab4ca21183e069e7d8b2dc68006594f40bddf1b15bbd  0.0s
    "LastUpdateStatusReasonCode": "Creating",
 => [internal] load build definition from Dockerfile                                                             0.0s
 => [internal] load .dockerignore                                                                                0.0s
 => [internal] load metadata for public.ecr.aws/lambda/provided:al2                                              1.1s
 => CACHED [build 1/8] FROM public.ecr.aws/lambda/provided:al2@sha256:33ab84899549495234fd55839a00e1baf9406e80b  0.0s
 => [internal] load build context                                                                                0.2s
 => => transferring context: 5.43kB                                                                              0.1s
 => CACHED [build 2/8] RUN yum install -y golang                                                                 0.0s
 => CACHED [build 3/8] RUN go env -w GOPROXY=direct                                                              0.0s
 => CACHED [build 4/8] WORKDIR /app                                                                              0.0s
 => CACHED [build 5/8] COPY go.mod go.sum ./                                                                     0.0s
 => CACHED [build 6/8] RUN go mod download                                                                       0.0s
 => [build 7/8] COPY *.go ./                                                                                     0.2s
 => [build 8/8] RUN go build -o /app/main                                                                      177.7s  <----
 => [stage-1 2/2] COPY --from=build /app/main /main                                                              0.9s
 => exporting to image                                                                                           0.5s
 => => exporting layers                                                                                          0.4s
 => => writing image sha256:98a228a29c7b05c144ba2ade6b61d55b414af24da8029342042bd55c48a432e7

Steps to Reproduce the Problem

Create vanilla Lambda using the base public.ecr.aws/lambda/provided:al2 image. Measure build time of image.
Add this package as a dep, wrap the handler. Measure build time of image.

Specifications

Datadog Go Lambda package version: v1.6.0
Go version: 1.18

Stacktrace

Allow to set variables via datadog.yaml

Expected Behavior

Would be cool if I can use

DD_TRACE_ENABLED
DD_TRACE_STARTUP_LOGS
DD_VERSION
DD_UNIVERSAL_INSTRUMENTATION

via datadog.yaml config for my lambdas - now I have 90% of DD configuration there but these if I am correct are not supported.

For me as a user is not clear and correct why DD_ENV could be move to yaml as env and is working but DD_VERSION can not be used as version there. Also second example DD_CAPTURE_LAMBDA_PAYLOAD works as capture_lambda_payload but DD_UNIVERSAL_INSTRUMENTATION does not

Also variable site - this is not clear for me:

datadog-lambda-go has in config option to set site - but I can define it via datadog.yaml if I am using extension. Am I correct that if I have extension then does not matter what I will define in datadog-lambda-go site parameter? Or how it works?

Specifications

Datadog Go Lambda package version: 1.18.0
Extension v35
Go version: 1.18

Support `lambda.StartHandler`.

Expected Behavior

DD only supports lambda.Start() which limits some of the functionality and uses of lambda. The difference between lambda.Start() and lambda.StartHandler() is that the former takes care of the serialization while the second operates and returns a []byte and accepts a struct that impliments Invoke()

For teams that have many lambdas that implement lambda.StartHandler() the lift to use DD is harder than if DD just supported that call behavior as well its an easier transition for teams.

Upgrade to github.com/aws/aws-sdk-go-v2

Please consider upgrading to v2 of the AWS SDK, because v1 is in maintenance mode and won't receive many handy features like the ability to unwrap errors (aws/aws-sdk-go#2820)

panic: runtime error: invalid memory address or nil pointer dereference

Expected Behavior

Nothing inside this module panics even when called by multiple goroutines.

Actual Behavior

We see panics periodically in lambda functions that invoked very frequently.

Steps to Reproduce the Problem

Run a lambda function that is invoked very frequently and normally re-uses execution contexts
Write some metrics via MetricWithTimestamp on a timer

Specifications

Datadog Go Lambda package version: 0.7.0
Go version: 1.14.6

Stacktrace

/go/pkg/mod/github.com/launchdarkly/[email protected]/fmetrics/metrics.go:161 +0x500
-- | -- | -- | -- | --
created by github.com/launchdarkly/foundation/fmetrics.Initialize
  /go/pkg/mod/github.com/launchdarkly/[email protected]/fmetrics/metrics.go:173 +0x59
 github.com/launchdarkly/foundation/fmetrics.Initialize.func1(0x1853aa0, 0xc00003a2e0, 0x0, 0x0, 0x0, 0x0, 0x7d3, 0x0, 0x0, 0xc00003a03a, ...)
/go/pkg/mod/github.com/launchdarkly/[email protected]/fmetrics/metrics.go:247 +0x6e
github.com/launchdarkly/foundation/fmetrics.Flush()
/go/pkg/mod/github.com/launchdarkly/[email protected]/fmetrics/ddlambda/reporter.go:110 +0x68
 github.com/launchdarkly/foundation/fmetrics/ddlambda.(*Reporter).FlushOnce(0xc000100500, 0x0, 0x0)
/go/pkg/mod/github.com/launchdarkly/[email protected]/registry.go:65 +0xb8
github.com/launchdarkly/go-metrics.(*StandardRegistry).Each(0xc0002e73f0, 0xc004d43ff0)
/go/pkg/mod/github.com/launchdarkly/[email protected]/fmetrics/ddlambda/reporter.go:156 +0xa12
 github.com/launchdarkly/foundation/fmetrics/ddlambda.(*Reporter).FlushOnce.func1(0xc0000371e0, 0x1a, 0x1498840, 0xc000088240)
 /go/pkg/mod/github.com/!data!dog/[email protected]/ddlambda.go:114 +0x84
 github.com/DataDog/datadog-lambda-go.Metric(0xc00abbacc0, 0x23, 0x0, 0x0, 0x0, 0x0)
/go/pkg/mod/github.com/!data!dog/[email protected]/ddlambda.go:126 +0x5a
 github.com/DataDog/datadog-lambda-go.MetricWithTimestamp(0xc00abbacc0, 0x23, 0x0, 0xbfc809c2883dcee6, 0x728092c5fb6, 0x2273140, 0x0, 0x0, 0x0)
 /go/pkg/mod/github.com/!data!dog/[email protected]/internal/metrics/context.go:19
 github.com/DataDog/datadog-lambda-go/internal/metrics.GetListener(...)
  /usr/local/go/src/context/context.go:550 +0x26
  context.(*valueCtx).Value(0x0, 0x12196a0, 0xc00011cac8, 0x83dcee6, 0x83dcee600000006)
 goroutine 10 [running]:
  [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x495dd6]
  panic: runtime error: invalid memory address or nil pointer dereference

Based on a brief review of the code involved in the stack trace, it appears that there's a data race in this module. GetContext can be called by any goroutine and it accesses wrapper.CurrentContext. wrapper.CurrentContext is written here without any locking to guard against concurrent access. Since our metric reporting is run on a timer, it can happen concurrently with the wrapper's cleanup code.

How can I send custom non-distribution metrics?

The Readme says:

Custom metrics can be submitted using the Metric function. The metrics are submitted as distribution metrics.

Is there anyway to send other kinds of metrics? Right now I want to send count metrics from AWS Lambda to Datadog. How might I go about doing that?

datadog / datadog-lambda-go Goto Github PK

datadog-lambda-go's Introduction

datadog-lambda-go

Installation

Configurations

Opening Issues

Contributing

Community

License

datadog-lambda-go's People

Contributors

Stargazers

Watchers

Forkers

datadog-lambda-go's Issues

Expected Behavior

Actual Behavior

Specifications

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Expected Behavior

Expected Behavior

Actual Behavior

Specifications

Expected Behavior

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

Stacktrace

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

Stacktrace

Expected Behavior

Actual Behavior

Specifications

Stacktrace

Expected Behavior

Actual Behavior

Specifications

Expected Behavior

Actual Behavior

Specifications

Stacktrace

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

Stacktrace

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

Stacktrace

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Specifications

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem