laurmurclar / datadog-to-terraform Goto Github PK

View Code? Open in Web Editor NEW

235.0 235.0 39.0 525 KB

Converts Datadog resource JSON into Terraform alarm code.

Home Page: https://chrome.google.com/webstore/detail/datadog-to-terraform-conv/lafmglpipgongjmbbjngmboifpaodemk

License: MIT License

JavaScript 93.78% HTML 6.22%

browser-extension chrome-extension datadog firefox-addon hacktoberfest terraform terraform-provider

datadog-to-terraform's People

Contributors

Stargazers

Watchers

datadog-to-terraform's Issues

Not able to convert the new keywords which is defined in datadog json such as formulas, response_type, name

Describe the bug
As per the new datadog dashboard, few attributes are not able to convert by the datadog converter plugin

To Reproduce
Use the below JSON to reproduce the problem

{ "title": "An example screenboard which causes the issue" }

{
"title": "test.dev.edit",
"description": "",
"widgets": [
{
"id": 6724092998888202,
"layout": {
"x": 8,
"y": 20,
"width": 8,
"height": 6
},
"definition": {
"title_size": "16",
"title_align": "left",
"type": "query_value",
"requests": [
{
"formulas": [
{
"formula": "query1"
}
],
"response_format": "scalar",
"queries": [
{
"query": "avg:test.stats-cancellation}.rollup(86400)",
"data_source": "metrics",
"name": "query1",
"aggregator": "last"
}
]
}
],
"autoscale": false,
"precision": 0
}
}
],
"template_variables": [
{
"name": "DashEnv",
"default": "prod2",
"prefix": "environment"
}
],
"layout_type": "free",
"is_read_only": false,
"notify_list": [],
"id": "a9e-yvm-s3z"
}
Expected terraform code
It should be able to convert.

resource "datadog_dashboard" "not broken, woo!" { ... }

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Support Datadog Timeboards

Is your feature request related to a problem? Please describe.
In addition to monitor conversion, support Timeboard conversion.

Describe the solution you'd like
Automatically detect monitor or timeboard format and output appropriate terraform.

Conversion for "template_variable_presets" not found

Copy the following json output from Datadog:

Clicking convert, gives the following error:
Conversion for "template_variable_presets" not found

❗️Conversion for "priority" not found

Describe the bug
I tried to configure a monitor in Datadog and use this plugin in firefox to convert to terraform. Here was the JSON I copied from Datadog's export:

{
	"id": 30072632,
	"name": "Concurrent lambda invocations >80% of unreserved concurrency limit",
	"type": "metric alert",
	"query": "max(last_15m):avg:aws.lambda.unreserved_concurrent_executions.maximum{account:xxxx} > 800",
	"message": "The number of unreserved concurrent Lambda executions has exceeded 80% of our account limit in {{account}}. If the limit is reached Lambda executions will start to be throttled by AWS. We should determine if there's a function being executed more often than expected or if we should request a limit increase for the account. ",
	"tags": [],
	"options": {
		"notify_audit": true,
		"locked": false,
		"timeout_h": 0,
		"new_host_delay": 300,
		"require_full_window": true,
		"notify_no_data": false,
		"renotify_interval": "0",
		"evaluation_delay": 300,
		"escalation_message": "",
		"no_data_timeframe": null,
		"include_tags": true,
		"thresholds": {
			"critical": 800
		}
	},
	"priority": 1
}

When I tried to convert, I received the following error message: ❗️Conversion for "priority" not found

If I remove the priority key from the JSON the HCL is generated correctly.

It seems like this might be a new property added to Datadog since this extension was last released? It looks like it is supported in the terraform provider:

priority (Number)

Expected behavior
I expect that this property would be supported and generate the correct terraform configuration.

Screenshots

System info (please complete the following information):

OS: MacOs Big Sur
Browser: Firefox
Version: 84.0.2

Testing

Would be great to have testing to make changes safer, especially as more people contribute

Can't convert key 'on_right_yaxis'

I'm using version 3.5.0 of the Datadog-to-Terraform Converter Chrome extension.

Trying the convert a Datadog dashboard JSON fails with error:

Can't convert key 'on_right_yaxis'

The JSON is following:

{
  "title": "Title",
  "description": "Description",
  "widgets": [
    {
      "id": 6094222782053076,
      "definition": {
        "title": "ECS",
        "type": "group",
        "layout_type": "ordered",
        "widgets": [
          {
            "id": 2061680356847030,
            "definition": {
              "title": "Num Services Running",
              "show_legend": false,
              "legend_size": "0",
              "type": "timeseries",
              "requests": [
                {
                  "q": "sum:aws.ecs.service.running{$env,$client} by {servicename}",
                  "on_right_yaxis": false,
                  "style": {
                    "palette": "dog_classic",
                    "line_type": "solid",
                    "line_width": "normal"
                  },
                  "display_type": "area"
                }
              ],
              "yaxis": {
                "scale": "linear",
                "include_zero": true,
                "min": "auto",
                "max": "auto"
              }
            }
          }
        ]
      }
    }
  ],
  "template_variables": [
    { "name": "env", "default": "*", "prefix": "env" },
    { "name": "client", "default": "*", "prefix": "client" }
  ],
  "layout_type": "ordered",
  "is_read_only": true,
  "notify_list": [],
  "id": "dru-pjw-8zd"
}

Error handling

We don't show errors in the extension UI, which can be confusing.

It would be nice to show parsing errors like Could not transform key 'threshold_windows'

Error - Can't convert key `'tags'`

Describe the bug
Error - Can't convert key 'tags'

To Reproduce
Some full example input which reproduces the bug, eg.

{
  "title": "Lightspeed CloudFront",
  "description": "[[suggested_dashboards]]",
  "widgets": [
    {
      "id": 2521798555330034,
      "layout": {
        "x": 0.08333333333333333,
        "y": 0.08333333333333333,
        "width": 24,
        "height": 6
      },
      "definition": {
        "type": "free_text",
        "text": "Lightspeed CloudFront",
        "color": "#2B13E6",
        "font_size": "auto",
        "text_align": "left"
      }
    }
  ],
  "layout_type": "free",
  "notify_list": [],
  "tags": [
    "team:lightspeed-product-api"
  ]
}

Expected terraform code

resource "datadog_dashboard" "not broken, woo!" { ... }

Screenshots

Additional context
If I remove this, the error does not occur:

  "tags": [
    "team:lightspeed-product-api"
  ]

Add a way to add defaults

We have some defaults that it would be good to set regardless of what gets pasted in.

locked set to true
Add tag "terraform:true" in the tags list if it doesn't already have it
Add #[MANAGED BY TERRAFORM] to the message at the bottom if it doesn't already have it

Anomaly detection monitors are not supported

Looks like the tool doesn't parse JSON objects for Anomaly Detection monitors.
It works with a standard metric alarm containing the same query though.

I believe this is happening Anomaly Detection has a new set of keys apart from thresholds that looks like this:

		"threshold_windows": {
			"trigger_window": "last_1h",
			"recovery_window": "last_30m"
		}

This do not seems to be handled.

To Conversion fails with: "Conversion for "id" not found.

when carrying out the process indicated in the documentation of copying the JSON content of a monitor, and putting the name of the resource, it does not generate the Terraform code with the indicated error.
❗️Conversion for "id" not found

Add support for other browsers

Add support for legend_columns & legend_layout

Add support to dashboard properties legend_columns & legend_layout when those are add to datadog provider (2.24.0 ?)
DataDog/terraform-provider-datadog#992

Dashboard Conversion Seeing Error TypeError: array.forEach is not a function

Describe the bug
Trying to convert a dashboard json to terraform is seeing a TypeError: array.forEach is not a function

To Reproduce
Try converting this dashboard

{
  "title": "bpham-test",
  "description": "",
  "widgets": [
    {
      "id": 3383947803306764,
      "definition": {
        "title": "k8s metrics",
        "type": "group",
        "layout_type": "ordered",
        "widgets": [
          {
            "id": 643181470432836,
            "definition": {
              "title": "CPU Usage per Node",
              "title_size": "16",
              "title_align": "left",
              "type": "hostmap",
              "requests": {
                "fill": {
                  "q": "sum:kubernetes.cpu.usage.total{$cluster_project_id,$cluster} by {host}"
                }
              },
              "no_metric_hosts": false,
              "no_group_hosts": true,
              "group": ["kube_metadata.node.label.cloud.google.com/gke-nodepool"],
              "scope": ["$project_id", "$cluster"],
              "style": { "palette": "YlOrRd", "palette_flip": false }
            }
          }
        ]
      }
    }
  ],
  "template_variables": [
    { "name": "cluster_project_id", "default": "testporo", "prefix": "project" },
    { "name": "cluster", "default": "ucr-cluster", "prefix": "cluster_name" },
    { "name": "namespace", "default": "*", "prefix": "kube_namespace" },
    { "name": "subscription_project_id", "default": "cluster", "prefix": "project_id" }
  ],
  "layout_type": "ordered",
  "is_read_only": false,
  "notify_list": [],
  "id": "mue-kck-jmi"
}

Expected behavior
Conversion should work.

Screenshots

System info (please complete the following information):

OS: Mac OS 11.2.2
Browser Chrome
Version 88.0.4324.192

Omit deprecated param 'logset'

Don't have permissions to create a branch:

app\dashboard-converter.js :
logset: (_) => "", // 2.23.0 deprecated, see docs for widget.log_stream_definition

examples\screenboard.json :

{
     "id": 1177803908467722,
     "layout": {
       "x": 968,
       "y": -242,
       "width": 33,
       "height": 21
     },
     "definition": {
       "title": "Plugins",
       "title_size": "13",
       "title_align": "left",
       "type": "log_stream",
       "logset": "",
       "indexes": [],
       "query": "$dd_env $app_group status:(error OR emergency)",
       "sort": {
         "column": "service",
         "order": "desc"
       },
       "show_date_column": true,
       "show_message_column": true,
       "message_display": "expanded-md"
     }

Support for Terraform 0.12.x

Not sure what needs to be done here, but marking it so I don't forget

Cannot convert new fields

Describe the bug
The plugin cannot convert these fields in DD monitor JSON:

restriction_query
on_missing_data
restricted_roles

To Reproduce

{
	"id": 10000000000,
	"name": "Console: A high error rate",
	"type": "rum alert",
	"query": "rum(\"-env:dev @error.source:(console OR source) @type:error\").rollup(\"count\").last(\"15m\") > 3",
	"message": "A high error rate",
	"tags": [
		"env:prod"
	],
	"options": {
		"thresholds": {
			"critical": 3
		},
		"enable_logs_sample": false,
		"notify_audit": false,
		"restriction_query": null,
		"on_missing_data": "default",
		"include_tags": false,
		"new_host_delay": 300,
		"groupby_simple_monitor": false,
		"silenced": {}
	},
	"priority": 2,
	"restricted_roles": [
		"ROLE_ID"
	]
}

Screenshots

Errors converting Monitor

❗️Can't convert key 'queryConfig'
❗️Can't convert key 'aggregation'

{
	"id": 32168291,
	"name": "DueDiligence - KYC - [Non-Prod] - Errors on Logs",
	"type": "log alert",
	"query": "logs(\"source:due_diligence env:(dev OR qa OR sbox) appgroup:dd-kyc status:(error OR emergency)\").index(\"*\").rollup(\"count\").by(\"service\").last(\"15m\") >= 1",
	"message": "{{#is_alert}}:notify @slack-DueDiligence-dd-dev-logs{{/is_alert}}",
	"tags": [
		"appgroup:dd-kyc",
		"duediligence",
		"log-alert",
		"monitor_env:non-prod",
		"terraform"
	],
	"options": {
		"include_tags": true,
		"notify_audit": false,
		"locked": true,
		"timeout_h": 0,
		"silenced": {},
		"enable_logs_sample": true,
		"thresholds": {
			"critical": 1
		},
		"new_host_delay": 300,
		"require_full_window": true,
		"notify_no_data": false,
		"renotify_interval": 0,
		"groupby_simple_monitor": true,
		"queryConfig": {
			"logset": {
				"name": "*"
			},
			"track": "logs",
			"timeRange": {
				"to": 1616759698501,
				"live": true,
				"from": 1616745298501
			},
			"queryString": "source:due_diligence AND env:(dev OR qa OR sbox) AND appgroup:dd-kyc AND status:(error OR emergency)",
			"indexes": [
				"*"
			],
			"queryIsFailed": false
		},
		"aggregation": {
			"metric": "count",
			"type": "count",
			"groupBy": "core_service"
		}
	},
	"priority": 4
}

Add Monitor - "classification" property

New property "classification" for every monitor
values e.g. 'log', 'metric', ...

Automated publishing is broken

Describe the bug
Usually admins publish the extension by tagging a new release on Github. That kicks off a CircleCI workflow which bundles up the extension and publishes it to the Chrome Store automatically.

I did that for the latest release (v1.3) and the change never hit the Chrome Store. We should look into the Circle build and see if it actually passed. I manually uploaded the new version to the store, so we're not entirely blocked on shipping

Terraform code generation fails if `threshold_windows` present in monitor JSON

Converting JSON to Terraform code failed when I tried it with this JSON:

{
	"name": "Inbound Email Rejection: App Not Found",
	"type": "metric alert",
	"query": "avg(last_1h):anomalies(sum:<metric>.as_count(), 'agile', 2, direction='both', alert_window='last_5m', interval=20, count_default_zero='true', seasonality='hourly') >= 1",
	"message": "Blah",
	"tags": [],
	"options": {
		"notify_audit": false,
		"locked": false,
		"timeout_h": 0,
		"new_host_delay": 300,
		"require_full_window": true,
		"notify_no_data": true,
		"renotify_interval": "0",
		"escalation_message": "",
		"no_data_timeframe": 10,
		"include_tags": false,
		"thresholds": {
			"critical": 1,
			"critical_recovery": 0
		},
		"threshold_windows": {
			"trigger_window": "last_5m",
			"recovery_window": "last_15m"
		}
	}
}

@laurmurclar helped me and after removing

"threshold_windows": {
  "trigger_window": "last_5m",
  "recovery_window": "last_15m"
}

from the options sub-object it worked.

Terraform apply fails due to blocks being defined as arguments during conversion from json

Describe the bug
Screenboard or Timeboard JSON converted into Terraform/HCL fails terraform apply due to incorrect format of style, yaxis, custom_links attribute blocks (maybe others). I believe this is a more recent change in terraform, using terraform 13.5 at the moment.

To Reproduce

Create a dashboard with a time series graph
Export json and paste into extension
Put into terraform file and apply
See failures similiar to below:

Error: Unsupported argument

  on datadog.tf line 40, in resource "datadog_dashboard" "ay-dashboard-test":
  40:       yaxis = {

An argument named "yaxis" is not expected here. Did you mean to define a block
of type "yaxis"?


Error: Unsupported argument

  on datadog.tf line 67, in resource "datadog_dashboard" "ay-dashboard-test":
  67:         style = {

An argument named "style" is not expected here. Did you mean to define a block
of type "style"?

Expected behavior
Terraform apply should create dashboard

System info (please complete the following information):

Browser: Happened on both Firefox and Chrome extensions
Version: 3.4.1
Terraform version: 13.5

Additional context
As a workaround search and replace in the converted terraform output

yaxis = { to yaxis {
style = { to style {
custom_links I just deleted since we aren't using them

Screenboard - incorrect log_query

Converting a widget doesn't return the correct query.
Instead we get:
log_query = [object Object]

Format output

Currently, the output is not formatted and you need to run terraform fmt on it before using. You can make this easier by having your text editor run it on save of all terraform files, but it would be nice to have this built in to the extension.

Screenboard - can't convert several keys

Screenboard, missing to convert definition key properties:

"type": "manage_status",

count
start

"type": "note",

vertical_align
has_padding

❗️Can't convert key 'logset'

Safari Support

Is your feature request related to a problem? Please describe.
I have to switch to a different browser to use this extension.

Describe the solution you'd like
It would be great if I could install this extension on Safari.

Describe alternatives you've considered
N/A

Additional context
N/A

Can't convert key `show_title`

Describe the bug
Cant convert key show title

To Reproduce
Some full example input which reproduces the bug, eg.

  "widgets": [
    {
      "id": 5474453034391576,
      "definition": {
        "title": "Errors",
        "show_title": true,
        "type": "group",
        "layout_type": "ordered",
        "widgets": [

Screenshots

Upgrade to Manifest V3

The manifest_version we use (in manifest.json) is being phased out and we need to upgrade it to V3. Not sure what is involved but here are some links I found with 5 secs of googling:

Cannot convert new Datadog json fields

Describe the bug
The Datadog dashboard json format has been changed, there are new fields, for example: show_title, queries, etc.

To Reproduce
Create a new dashboard on Datadog UI and convert it.

{
   "title":"Title",
   "description":"",
   "widgets":[ 
          {
            "definition":{
               "title":"Title",
               "title_size":"16",
               "title_align":"left",
               "show_legend":true,
               "legend_layout":"auto",
               "legend_columns":[
                  "value"
               ],
               "time":{
                  
               },
               "type":"timeseries",
               "requests":[
                  {
                     "queries":[
                        {
                           "data_source":"metrics",
                           "name":"query1",
                           "query":"metric1{$account}"
                        },
                        {
                           "data_source":"metrics",
                           "name":"query2",
                           "query":"metric2{$account}"
                        }
                     ],
                     "response_format":"timeseries",
                     "style":{
                        "palette":"dog_classic",
                        "line_type":"solid",
                        "line_width":"normal"
                     },
                     "display_type":"line"
                  }
               ],
               "yaxis":{
                  "include_zero":false
               },
               "markers":[
                  
               ]
            },
            "layout":{
               "x":4,
               "y":0,
               "width":4,
               "height":2
            }
          }
       ],
   "template_variables":[
      {
         "name":"account",
         "default":"account1",
         "prefix":"account"
      }
   ],
   "layout_type":"ordered",
   "is_read_only":false,
   "notify_list":[
      
   ],
   "reflow_type":"fixed"
}

The error message from Chrome extension: ❗️Can't convert key 'queries'

Add an "Export to Terraform" button into the Datadog UI

Chrome Extensions can modify the DOM - it would be awesome to have a button in the Export Monitor panel in Datadog. The button could take care of pulling out the monitor json & doing code generation all in one step.

Errors converting Monitor

Can't convert key 'groupby_simple_monitor'
Can't convert key 'silenced'

{
	"id": 123456789,
	"name": "DueDiligence - KYC - 422's on Logs",
	"type": "log alert",
	"query": "logs(\" source:due_diligence AND env:prod AND appgroup:dd-kyc AND @Properties.StatusCode:422\").index(\"*\").rollup(\"count\").by(\"service\").last(\"1d\") >= 5",
	"message": ":notify @slack-DueDiligence-dd-prod-warning-logs",
	"tags": [
		"appgroup:dd-kyc",
		"duediligence",
		"log-alert",
		"monitor_env:prod",
		"terraform"
	],
	"options": {
		"notify_audit": false,
		"locked": true,
		"silenced": {},
		"enable_logs_sample": true,
		"thresholds": {
			"critical": 5
		},
		"new_host_delay": 300,
		"require_full_window": true,
		"notify_no_data": false,
		"include_tags": true,
		"groupby_simple_monitor": true
	},
	"priority": 3
}

laurmurclar / datadog-to-terraform Goto Github PK

datadog-to-terraform's People

Contributors

Stargazers

Watchers

Forkers

datadog-to-terraform's Issues

Recommend Projects

Recommend Topics

Recommend Org