Code Monkey home page Code Monkey logo

terraform-ibm-observability-agents's Introduction

Terraform IBM Observability agents module

Graduated (Supported) pre-commit latest release Renovate enabled semantic-release

This module deploys the following observability agents to a Red Hat OpenShift Container Platform or Kubernetes cluster:

Overview

terraform-ibm-observability-agents

Usage

# ############################################################################
# Init cluster config for helm
# ############################################################################

data "ibm_container_cluster_config" "cluster_config" {
  # update this value with the Id of the cluster where these agents will be provisioned
  cluster_name_id = "cluster_id"
}

# ############################################################################
# Config providers
# ############################################################################

provider "ibm" {
  # update this value with your IBM Cloud API key value
  ibmcloud_api_key = "api key value"  # pragma: allowlist secret
}

provider "helm" {
  kubernetes {
    host                   = data.ibm_container_cluster_config.cluster_config.host
    token                  = data.ibm_container_cluster_config.cluster_config.token
    cluster_ca_certificate = data.ibm_container_cluster_config.cluster_config.ca_certificate
  }
}

# ############################################################################
# Install observability agents
# ############################################################################

module "observability_agents" {
  source                           = "terraform-ibm-modules/observability-agents/ibm"
  version                          = "X.X.X" # Replace "X.X.X" with a release version to lock into a specific release
  cluster_id                       = "cluster id" # update this with your cluster id where the agents will be installed
  cluster_resource_group_id        = "resource group id" # update this with the Id of your IBM Cloud resource group
  log_analysis_ingestion_key       = "XXXXXXXX"
  log_analysis_instance_region     = "us-south"
  cloud_monitoring_access_key      = "XXXXXXXX"
  cloud_monitoring_instance_region = "us-south"
}

Configuration for Kubernetes metadata filtering in the logging agent

You can configure the logging agent to filter log lines according to the Kubernetes resources metadata by setting the exclusion and inclusion parameters.

For example, to set the agent to return all log lines coming from the default Kubernetes namespace and exclude anything with a label app.kubernetes.io/name with value sample-app or an annotation annotation.user with value sample-user, include these parameters:

custom_log_analysis_at_agent_line_exclusion = "label.app.kubernetes.io/name:sample-app\\, annotation.user:sample-user"
custom_log_analysis_at_agent_line_inclusion = "namespace:default"

The following is the corresponding DaemonSet configuration:

- name: LOGDNA_K8S_METADATA_LINE_INCLUSION
  value: "label.app.kubernetes.io/name:sample-app, annotation.user:sample-user"
- name: LOGDNA_K8S_METADATA_LINE_EXCLUSION
  value: "namespace:default"

For more information, see Configuration for Kubernetes Metadata Filtering.

Required IAM access policies

You need the following permissions to run this module.

  • IAM Services
    • IBM Cloud Activity Tracker service
      • Viewer platform access
      • Reader service access
    • IBM Cloud Monitoring service
      • Viewer platform access
      • Reader service access
    • IBM Log Analysis service
      • Viewer platform access
      • Reader service access
    • Kubernetes service
      • Viewer platform access
      • Manager service access

Requirements

Name Version
terraform >= 1.1.0, <1.7.0
helm >= 2.8.0, <3.0.0
ibm >= 1.59.0, <2.0.0

Modules

No modules.

Resources

Name Type
helm_release.cloud_monitoring_agent resource
helm_release.log_analysis_agent resource
ibm_container_cluster_config.cluster_config data source
ibm_container_vpc_cluster.cluster data source

Inputs

Name Description Type Default Required
cloud_monitoring_access_key Access key used by the IBM Cloud Monitoring agent to communicate with the instance string null no
cloud_monitoring_add_cluster_name If true, configure the cloud monitoring agent to attach a tag containing the cluster name to all metric data. bool true no
cloud_monitoring_agent_name Cloud Monitoring agent name. Used for naming all kubernetes and helm resources on the cluster. string "sysdig-agent" no
cloud_monitoring_agent_namespace Namespace where to deploy the Cloud Monitoring agent. Default value is 'ibm-observe' string "ibm-observe" no
cloud_monitoring_agent_tags List of tags to associate to all matrics that the agent collects. NOTE: Use the 'cloud_monitoring_add_cluster_name' variable to add the cluster name as a tag. list(string) [] no
cloud_monitoring_agent_tolerations List of tolerations to apply to Cloud Monitoring agent. 
list(object({
    key               = optional(string)
    operator          = optional(string)
    value             = optional(string)
    effect            = optional(string)
    tolerationSeconds = optional(number)
  }))
 
[
  {
    "operator": "Exists"
  },
  {
    "effect": "NoSchedule",
    "key": "node-role.kubernetes.io/master",
    "operator": "Exists"
  }
]
 no
cloud_monitoring_enabled Deploy IBM Cloud Monitoring agent bool true no
cloud_monitoring_endpoint_type Specify the IBM Cloud Monitoring instance endpoint type (public or private) to use. Used to construct the ingestion endpoint. string "private" no
cloud_monitoring_instance_region The IBM Cloud Monitoring instance region. Used to construct the ingestion endpoint. string null no
cloud_monitoring_metrics_filter To filter custom metrics, specify the Cloud Monitoring metrics to include or to exclude. See https://cloud.ibm.com/docs/monitoring?topic=monitoring-change_kube_agent#change_kube_agent_inc_exc_metrics. 
list(object({
    type = string
    name = string
  }))
 [] no
cloud_monitoring_secret_name The name of the secret which will store the access key. string "sysdig-agent" no
cluster_config_endpoint_type Specify which type of endpoint to use for for cluster config access: 'default', 'private', 'vpe', 'link'. 'default' value will use the default endpoint of the cluster. string "default" no
cluster_id The ID of the cluster you wish to deploy the agents in string n/a yes
cluster_resource_group_id The Resource Group ID of the cluster string n/a yes
log_analysis_add_cluster_name If true, configure the log analysis agent to attach a tag containing the cluster name to all log messages. bool true no
log_analysis_agent_custom_line_exclusion Log Analysis agent custom configuration for line exclusion setting LOGDNA_K8S_METADATA_LINE_EXCLUSION. See https://github.com/logdna/logdna-agent-v2/blob/master/docs/KUBERNETES.md#configuration-for-kubernetes-metadata-filtering for more info. string null no
log_analysis_agent_custom_line_inclusion Log Analysis agent custom configuration for line inclusion setting LOGDNA_K8S_METADATA_LINE_INCLUSION. See https://github.com/logdna/logdna-agent-v2/blob/master/docs/KUBERNETES.md#configuration-for-kubernetes-metadata-filtering for more info. string null no
log_analysis_agent_name Log Analysis agent name. Used for naming all kubernetes and helm resources on the cluster. string "logdna-agent" no
log_analysis_agent_namespace Namespace where to deploy the Log Analysis agent. Default value is 'ibm-observe' string "ibm-observe" no
log_analysis_agent_tags List of tags to associate to all log records that the agent collects so that you can identify the agent's data quicker in the logging UI. NOTE: Use the 'log_analysis_add_cluster_name' variable to add the cluster name as a tag. list(string) [] no
log_analysis_agent_tolerations List of tolerations to apply to Log Analysis agent. 
list(object({
    key               = optional(string)
    operator          = optional(string)
    value             = optional(string)
    effect            = optional(string)
    tolerationSeconds = optional(number)
  }))
 
[
  {
    "operator": "Exists"
  }
]
 no
log_analysis_enabled Deploy IBM Cloud Logging agent bool true no
log_analysis_endpoint_type Specify the IBM Log Analysis instance endpoint type (public or private) to use. Used to construct the ingestion endpoint. string "private" no
log_analysis_ingestion_key Ingestion key for the IBM Cloud Logging agent to communicate with the instance string null no
log_analysis_instance_region The IBM Log Analysis instance region. Used to construct the ingestion endpoint. string null no
log_analysis_secret_name The name of the secret which will store the ingestion key. string "logdna-agent" no

Outputs

No outputs.

Contributing

You can report issues and request features for this module in GitHub issues in the module repo. See Report an issue or request a feature.

To set up your local development environment, see Local development setup in the project documentation.

terraform-ibm-observability-agents's People

Contributors

aashiq-j avatar ahmad-slo avatar akocbek avatar bhpratt avatar daniel-butler-irl avatar dishankkalra23 avatar jojustin avatar khuzaima05 avatar mounika-nalla avatar ocofaigh avatar rajatagarwal-ibm avatar terraform-ibm-modules-ops avatar toddgiguere avatar vbontempi avatar vburckhardt avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

toddgiguere berkcelc bhpratt

terraform-ibm-observability-agents's Issues

Support LogDNA/mezmo customisation for logs tags and exclusion rules

Description

In order to customise the logdna traces collected&pushed by the logdna agent deployed through the observability-agents-module module, implement the following customization options:

  • the list of tags metadata to attach to lines forwarded by the agent

this can be easily done by extending the logdna agent chart template at https://github.ibm.com/GoldenEye/observability-agents-module/blob/master/chart/logdna-agent/templates/daemonset.yaml
by adding something like

{{- if
.Values.logdna.tags
}}
          - name: LOGDNA_TAGS
            value: {{
.Values.logdna.tags
}}

in the template and by modifying the helm release module accordingly

  • the logdna exclusion rules to exclude log traces from the agent collection and/or from the logdna storage

this can be done with LOGDNA_LINE_EXCLUSION_REGEX env var configuration
More details on
https://github.com/logdna/logdna-agent-v2/blob/master/docs/README.md#configuring-regex-for-redaction-and-exclusion-or-inclusion
https://github.com/logdna/logdna-agent-v2/blob/master/docs/REGEX.md

Other details:
https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-log_analysis_agent

https://github.com/logdna/logdna-agent-v2/blob/master/docs/README.md

LogDNA api to implement exclusion rules on logdna instance (allows to stream the logs that are visible only in live mode)
https://docs.mezmo.com/log-analysis-api/ref#get-v1-config-exclusions

New or Affected Modules

observability-agents-module

By submitting this issue, you agree to follow our Code of Conduct

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.