Manage Datadog Monitors / Dashboards / Slos as code

• DRY, searchable, audited, documented
• Changes are PR reviewed and applied on merge
• Updating shows diff before applying
• Automated import of existing resources
• Resources are grouped into projects that belong to teams and inherit tags
• No copy-pasting of ids to create new resources
• Automated cleanup when removing code
• Helpers for automating common tasks

# teams/foo.rb
module Teams
  class Foo < Kennel::Models::Team
    defaults(mention: -> { "@slack-my-team" })
  end
end

# projects/bar.rb
class Bar < Kennel::Models::Project
  defaults(
    team: -> { Teams::Foo.new }, # use mention and tags from the team
    parts: -> {
      [
        Kennel::Models::Monitor.new(
          self, # the current project
          type: -> { "query alert" },
          kennel_id: -> { "load-too-high" }, # pick a unique name
          name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
          message: -> {
            <<~TEXT
              This is bad!
              #{super()} # inserts mention from team
            TEXT
          },
          query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" },
          critical: -> { 20 }
        )
      ]
    }
  )
end

• create a new private kennel repo for your organization (do not fork this repo)
• use the template folder as starting point:

  git clone [email protected]:your-org/kennel.git
  git clone [email protected]:grosser/kennel.git seed
  mv seed/template/* kennel/
  cd kennel && git add . && git commit -m 'initial'

• add a basic projects and teams so others can copy-paste to get started
• setup CI build for your repo (travis and Github Actions supported)
• uncomment .travis.yml section for datadog updates on merge (TODO: example setup for Github Actions)
• follow Setup in your repos Readme.md

• projects/ monitors/dashboards/etc scoped by project
• teams/ team definitions
• parts/ monitors/dashboards/etc that are used by multiple projects
• generated/ projects as json, to show current state and proposed changes in PRs

• mention is used for all team monitors via super()
• renotify_interval is used for all team monitors (defaults to 0 / off)
• tags is used for all team monitors/dashboards (defaults to team:<team-name>)

# teams/my_team.rb
module Teams
  class MyTeam < Kennel::Models::Team
    defaults(
      mention: -> { "@slack-my-team" }
    )
  end
end

• use datadog monitor UI to create a monitor
• see below

• use datadog monitor UI to find a monitor
• get the id from the url
• run URL='https://app.datadoghq.com/monitors/123' bundle exec rake kennel:import and copy the output
• import task also works with SLO alerts, e.g. URL='https://app.datadoghq.com/slo/edit/123abc456def123/alerts/789' bundle exec rake kennel:import
• find or create a project in projects/
• add the monitor to parts: [ list, for example:

# projects/my_project.rb
class MyProject < Kennel::Models::Project
  defaults(
    team: -> { Teams::MyTeam.new }, # use existing team or create new one in teams/
    parts: -> {
      [
        Kennel::Models::Monitor.new(
          self,
          id: -> { 123456 }, # id from datadog url, not necessary when creating a new monitor
          type: -> { "query alert" },
          kennel_id: -> { "load-too-high" }, # make up a unique name
          name: -> { "Foobar Load too high" }, # nice descriptive name that will show up in alerts and emails
          message: -> {
            # Explain what behavior to expect and how to fix the cause
            # Use #{super()} to add team notifications.
            <<~TEXT
              Foobar will be slow and that could cause Barfoo to go down.
              Add capacity or debug why it is suddenly slow.
              #{super()}
            TEXT
          },
          query: -> { "avg(last_5m):avg:system.load.5{hostgroup:api} by {pod} > #{critical}" }, # replace actual value with #{critical} to keep them in sync
          critical: -> { 20 }
        )
      ]
    }
  )
end

• run PROJECT=my_project bundle exec rake plan, an Update to the existing monitor should be shown (not Create / Delete)
• alternatively: bundle exec rake generate to only locally update the generated json files
• review changes then git commit
• make a PR ... get reviewed ... merge
• datadog is updated by CI

Remove the code that created the resource. The next update will delete it (see above for PR workflow).

• go to datadog dashboard UI and click on New Dashboard to create a dashboard
• see below

• go to datadog dashboard UI and click on New Dashboard to find a dashboard
• get the id from the url
• run URL='https://app.datadoghq.com/dashboard/bet-foo-bar' bundle exec rake kennel:import and copy the output
• find or create a project in projects/
• add a dashboard to parts: [ list, for example:

class MyProject < Kennel::Models::Project
  defaults(
    team: -> { Teams::MyTeam.new }, # use existing team or create new one in teams/
    parts: -> {
      [
        Kennel::Models::Dashboard.new(
          self,
          id: -> { "abc-def-ghi" }, # id from datadog url, not needed when creating a new dashboard
          title: -> { "My Dashboard" },
          description: -> { "Overview of foobar" },
          template_variables: -> { ["environment"] }, # see https://docs.datadoghq.com/api/?lang=ruby#timeboards
          kennel_id: -> { "overview-dashboard" }, # make up a unique name
          layout_type: -> { "ordered" },
          definitions: -> {
            [ # An array or arrays, each one is a graph in the dashboard, alternatively a hash for finer control
              [
                # title, viz, type, query, edit an existing graph and see the json definition
                "Graph name", "timeseries", "area", "sum:mystats.foobar{$environment}"
              ],
              [
                # queries can be an Array as well, this will generate multiple requests
                # for a single graph
                "Graph name", "timeseries", "area", ["sum:mystats.foobar{$environment}", "sum:mystats.success{$environment}"],
                # add events too ...
                events: [{q: "tags:foobar,deploy", tags_execution: "and"}]
              ]
            ]
          }
        )
      ]
    }
  )
end

Setting id makes kennel take over a manually created datadog resource. When manually creating to import, it is best to remove the id and delete the manually created resource.

When an id is set and the original resource is deleted, kennel will fail to update, removing the id will cause kennel to create a new resource in datadog.

Having many projects (and their sub-resources) can quickly get out of hand.

Use this class structure to keep things organized:

# projects/project_a/base.rb
module ProjectA
  class Base < Kennel::Models::Project
    defaults(
      kennel_id: -> { "project_a" },
      ...

# projects/project_a/monitors/foo_alert.rb
module ProjectA
  module Monitors
    class FooAlert < Kennel::Models::Monitor
      ...

Some validations might be too strict for your usecase or just wrong, please open an issue and to unblock use the validate: -> { false } option.

Link resources with their kennel_id in the format project kennel_id + : + resource kennel_id, this should be used to create dependent resources like monitor + slos, so they can be created in a single update and can be re-created if any of them is deleted.

• rebase on updated master to not undo other changes
• figure out project name by converting the class name to snake-case
• run PROJECT=foo bundle exec rake kennel:update_datadog to test changes for a single project (monitors: remove mentions while debugging to avoid alert spam)
  • use PROJECT=foo,bar,... for multiple projects

Add to parts/<folder>.

module Monitors
  class LoadTooHigh < Kennel::Models::Monitor
    defaults(
      name: -> { "#{project.name} load too high" },
      message: -> { "Shut it down!" },
      type: -> { "query alert" },
      query: -> { "avg(last_5m):avg:system.load.5{hostgroup:#{project.kennel_id}} by {pod} > #{critical}" }
    )
  end
end

Reuse it in multiple projects.

class Database < Kennel::Models::Project
  defaults(
    team: -> { Kennel::Models::Team.new(mention: -> { '@slack-foo' }, kennel_id: -> { 'foo' }) },
    parts: -> { [Monitors::LoadTooHigh.new(self, critical: -> { 13 })] }
  )
end

Run rake kennel:alerts TAG=service:my-service to see all un-muted alerts for a given datadog monitor tag.

rake kennel:validate_mentions should run as part of CI

rake kennel:dump > tmp/dump
cat tmp/dump | grep foo

focus on a single type: TYPE=monitors

Show full resources or just their urls by pattern:

rake kennel:dump_grep DUMP=tmp/dump PATTERN=foo URLS=true
https://foo.datadog.com/dasboard/123
https://foo.datadog.com/monitor/123

rake kennel:nodata TAG=team:foo

• Setting FORCE_GET_CACHE=true will cache all get requests, which makes benchmarking improvements more reliable.
• Setting STORE=false will make rake plan not update the files on disk and save a bit of time

rake play
cd template
rake plan

Then make changes to play around, do not commit changes and make sure to revert with a rake kennel:update_datadog after deleting everything.

To make changes via the UI, make a new free datadog account and use it's credentaisl instead.

Michael Grosser
[email protected]
License: MIT