Support for EKS & CodeBuild Log Sources about datadog-serverless-functions HOT 11 CLOSED

A possible better approach would be to use a named regex match to get the service from the log group, this would then take into account changes/additions to the services that support log groups. An example regex: \/aws\/(?<service>.*)\/.*

from datadog-serverless-functions.

I'm not too sure about using regex for service determination. But supporting eks and codebuild definitely makes sense. Would you like to contribute?

from datadog-serverless-functions.

Is there a list that contains which of the AWS integrations include automatic pipelines for log sources? Perhaps the right approach would be to keep the list in sync with that list. Basically, if an integration knows about the logs, set the source to the integration, if it doesn't, then leave the source as cloudwatch. Happy to help contribute based on the desired approach

from datadog-serverless-functions.

^ cc: @NBParis

from datadog-serverless-functions.

Hello @moserke ,

EKS

So EKS is not supported as a log source as the Logs that are collected are from the application running into the managed Kubernetes containers which can therefore be running any kind of application (java application, apache, ....).

Collecting logs in EKS today relies on the AWS agent that forward logs to Cloudwatch and then this function to Datadog but we are waiting for AWS to add support for Firelens in EKS as well so that you can benefit from the direct integration with Datadog.

I would invite you to add a 👍 on this issue.

CodeBuild

For Codebuild, that is definitely a legitimate request and we should indeed add it. It is indeed a grat idea of applying the regex \/aws\/(?<service>.*)\/.* but that might only work for logs from Cloudwatch and not from S3 buckets so we might need to make sure we do not break any behaviour there but that is definitely something that should be investigated.

List of service with automatic pipeline

The list of integration that have an automatic pipelines are actually the one listed there. As we keep updating the list as we add new automatic pipelines.

We are also working on open sourcing the list of integration pipelines and their definition which would give you more visibility over what is supported and what formats are covered.

Let me know if that clarifies the situation.

from datadog-serverless-functions.

Thanks that's great information. In regards to EKS, this is more specifically the control plane logs, the logs that customers don't have access to in the cluster itself. We do use log forwarding from the running applications in the cluster to datadog. But the control plane (k8s api, controller, scheduler, etc) go to log groups in the form of /aws/eks/[cluster_name]/cluster.

For the codebuild ones, I did add that key to the lambda we have running and that worked well.

from datadog-serverless-functions.

@moserke Do you mind submitting a PR for codebuild since you already have it working? Thanks! 🙇

from datadog-serverless-functions.

No problem, submitted. Sorry for the delay, got sidetracked with other things.

from datadog-serverless-functions.

Merged! Thanks for your contribution!

from datadog-serverless-functions.

What is the release cadence? That is, when will this be available to pull?

from datadog-serverless-functions.

Roughly speaking, about once a month in the past, we usually try to bundle small features & bug fixes together. We are working on defining a release cycle.

from datadog-serverless-functions.