[BUG] High memory consumption/High ram usage/memory leak caused by dd-tracer-go in serverless AWS lambda about dd-trace-go HOT 8 CLOSED

Hi @arturg-acoustic! Thanks for reaching out. The recommendation is correct, but I think that the place you chose to initialize the tracer isn't quite right.

Please, check our recommended usage for AWS Lambda with github.com/DataDog/datadog-lambda-go here: https://docs.datadoghq.com/serverless/aws_lambda/installation/go/?tab=custom

I hope this helps. Please reach again if you need more assistance.

from dd-trace-go.

this guide does not touch the subject of using tracer.Start(tracer.WithRuntimeMetrics())

You are right. Although this is our official recommendation for AWS Lambda, if you want to run dd-trace-go as part of a lambda, I still think that starting the tracer in the HandleRequest isn't the best approach.

According to AWS Lambda official docs, your code should include a main function where you can initialize the tracer, so all the requests handled by the lambda will reuse it. Like this:

package main

import (
	"context"
	"fmt"
	"github.com/aws/aws-lambda-go/lambda"
	"gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

type event struct {
	// ...
}

// skipping other functions used in the HandleRequest.

func HandleRequest(ctx context.Context, _ event) (event, error) {
	currentSpan, ctx := tracer.StartSpanFromContext(ctx, "HandleRequest")
	currentSpan.SetTag("service", "service-name")
	traceparent := getTraceparent(ctx, currentSpan)
	traceID, traceIdErr := getTraceIdFromTraceparent(traceparent)
	ctx = context.WithValue(ctx, "trace_id", traceID)
	defer currentSpan.Finish()
        doLogic()
	return whateverTheFunctionReturns()
}

func main() {
	tracer.Start(tracer.WithRuntimeMetrics())
	defer tracer.Stop()

	lambda.Start(HandleRequest)
}

Keep in mind that starting the trace will slightly increase your cold start, but it shouldn't be noticeable.

from dd-trace-go.

One day the service stopped sending traces to datadog. Might be a problem related to other external factors (lambda layer for example). Update resolved the issue.

Thank you. I agree that issue might be closed - I opened it mostly for informational value on the fix.

from dd-trace-go.

this guide does not touch the subject of using tracer.Start(tracer.WithRuntimeMetrics())

from dd-trace-go.

@darccio thank you for your reply. I did figure out that would help, but I would like to point out that before the update, the code worked perfectly. It would be a good idea to prevent users from shooting themselves in the foot / adding proper documentation for using tracer.Start() in datadog go docs.

Maybe you guys could investigate what caused the change?

from dd-trace-go.

@arturg-acoustic The docs assume that the tracer is initialized once, and the tracer itself expects to be a singleton.

I agree we could add a warning to make this more clear. Creating a tracer for each request is a code smell for us.

There are some things that are not clear so we can investigate:

• What was your previous dd-trace-go version?
• What this the first time you used tracer.WithRuntimeMetrics?

from dd-trace-go.

dd-trace-go I used previously was 1.54. It was used in March of 2023. It stopped working around Summer/Autumn, but did not cause any memory leaks until the update in March of 2024.

from dd-trace-go.

@arturg-acoustic What do you mean by "it stopped working"? The tracer stopped or only the runtime metrics? In any case, using the tracer in this way - creating it in each request - isn't a best practice.

Actually, calling tracer.Stop in that place was probably introducing latency, as stopping the tracer flushes all the spans in each request, instead of doing that asynchronously during the lambda lifespan.

I'll make sure the docs are clear. I'll also try to understand what went wrong, but one year of updates is fair amount of possibilities to consider.

As the issue at hand is solved, I'll proceed to close the issue in the meantime.

from dd-trace-go.