Сейчас по UTC - 1542799764

Есть вот такой триггер
https://moira.skbkontur.ru/trigger/VostokProductionConsumerMessagesCountTooLow

В нём вот такой таргет
aliasByNode(movingMax(vostok.prod.*.grouphost.vostok*.message_count,'6h'),2,4)

Если запросить это в carbonapi (https://graphite.skbkontur.ru/carbonapi/render?format=json&target=aliasByNode(movingMax(vostok.prod.*.grouphost.vostok04.message_count,'6h'),2,4)&from=1542797505), получим вот что

[
    {
        "target": "consumer-metric.vostok04",
        "datapoints": [...
        ]
    },
    {
        "target": "consumer-tracing.vostok04",
        "datapoints": [...
        ]
    }
]

Если же не указать время from, то прилетают куча левых точек (смотри datapoints-vostok04.json )

В логах карбонапи:

{
  "level": "INFO",
  "timestamp": "2018-11-21T14:18:11.478+0300",
  "logger": "access",
  "message": "request served",
  "data": {
    "handler": "render",
    "carbonapi_uuid": "b669a0da-52e8-4c17-bc33-f3a7b50290b7",
    "username": "devops",
    "url": "/render?format=json&from=1542798465&target=aliasByNode%28movingMax%28vostok.prod.%2A.grouphost.vostok%2A.message_count%2C%276h%27%29%2C2%2C4%29&until=1542799091",
    "peer_ip": "127.0.0.1",
    "host": "carbonapi",
    "format": "json",
    "use_cache": true,
    "request_method": "GET",
    "targets": [
      "aliasByNode(movingMax(vostok.prod.*.grouphost.vostok*.message_count,'6h'),2,4)"
    ],
    "cache_timeout": 60,
    "metrics": [
      "vostok.prod.*.grouphost.vostok*.message_count"
    ],
    "runtime": 0.070850722,
    "http_code": 200,
    "carbonzipper_response_size_bytes": 23950,
    "send_globs": true,
    "from": 1542798465,
    "until": 1542799091,
    "from_raw": "1542798465",
    "until_raw": "1542799091",
    "uri": "/render?format=json&from=1542798465&target=aliasByNode%28movingMax%28vostok.prod.%2A.grouphost.vostok%2A.message_count%2C%276h%27%29%2C2%2C4%29&until=1542799091",
    "from_cache": false,
    "zipper_requests": 2
  }
}

То есть время в CarbonAPI тоже правильное.

State триггера:

{
  "metrics": {
    "consumer-logs.vostok04": {
      "event_timestamp": 1538487834,
      "state": "OK",
      "suppressed": false,
      "timestamp": 1542721005,
      "value": 12985
    },
    "consumer-logs.vostok06": {
      "event_timestamp": 1538980093,
      "state": "OK",
      "suppressed": false,
      "timestamp": 1542798015,
      "value": 1231950
    },
    "consumer-metric.vostok04": {
      "event_timestamp": 1542795105,
      "state": "OK",
      "suppressed": false,
      "timestamp": 1542798015,
      "value": 106
    },
    "consumer-metric.vostok06": {
      "event_timestamp": 1538980093,
      "state": "OK",
      "suppressed": false,
      "timestamp": 1542798015,
      "value": 4426
    },
    "consumer-sentry.vostok04": {
      "event_timestamp": 1542720975,
      "state": "WARN",
      "suppressed": false,
      "timestamp": 1542721005,
      "value": 0,
      "maintenance": 1543401956
    },
    "consumer-sentry.vostok06": {
      "event_timestamp": 1538980093,
      "state": "OK",
      "suppressed": false,
      "timestamp": 1542798015,
      "value": 2559585
    },
    "consumer-tracing.vostok04": {
      "event_timestamp": 1542795225,
      "state": "OK",
      "suppressed": false,
      "timestamp": 1542798015,
      "value": 179
    },
    "consumer-tracing.vostok06": {
      "event_timestamp": 1540967175,
      "state": "OK",
      "suppressed": false,
      "timestamp": 1542798015,
      "value": 2585
    },
    "metrics-aggregator.vostok04": {
      "event_timestamp": 1542720855,
      "state": "WARN",
      "suppressed": false,
      "timestamp": 1542721005,
      "value": 0,
      "maintenance": 1543401958
    },
    "metrics-aggregator.vostok06": {
      "event_timestamp": 1540994235,
      "state": "OK",
      "suppressed": false,
      "timestamp": 1542798015,
      "value": 2585
    }
  },
  "score": 2,
  "state": "OK",
  "timestamp": 1542798105,
  "event_timestamp": 1538141709,
  "last_successful_check_timestamp": 1542798105,
  "trigger_id": "VostokProductionConsumerMessagesCountTooLow"
}

If you use telegram bot, you make sure that you use telebot listener only on one instance of Moira. For this we are using distributed lock and extend it every half-ttl time interval. But if we can not extend it while lock record not foul, another instance can register bot and start new listener. In this case we should stop listener on addled instance. But telebot does not support graceful shutdown of pooling goroutine.
Now i use dirty hack and try to register bot again, after fail to extend of lock record. In most cases this solution can help. But such situation is possible.

There are two types of triggers in Moira: simple mode and advanced mode.

In simple mode you can specify one target, and in advanced mode you can specify several targets. Web UI enforces this, but API does not. Therefore, using API you can create simple mode triggers with several targets.

Consider this trigger, created from API:

{
  "id": "GraphiteStoragesFreeSpace",
  "name": "Graphite storage free space low",
  "desc": "Graphite ClickHouse",
  "targets": [
    "aliasByNode(DevOps.system.graphite01.disk._mnt_data.gigabyte_percentfree, 2, 4)",
    "aliasByNode(DevOps.system.sd2-graphite01.disk._mnt_data.gigabyte_percentfree, 2, 4)",
    "aliasByNode(DevOps.system.bst-graphite01.disk.root.gigabyte_percentfree, 2, 4)",
    "aliasByNode(DevOps.system.dtl-graphite01.disk._mnt_data.gigabyte_percentfree, 2, 4)"
  ],
  "warn_value": 10,
  "error_value": 5,
  "trigger_type": "falling",
  "tags": [
    "Normal",
    "DevOps",
    "DevOpsGraphite-duty"
  ],
  "ttl_state": "NODATA",
  "ttl": 600,
  "sched": ...,
  "expression": "",
  "patterns": [
    "DevOps.system.graphite01.disk._mnt_data.gigabyte_percentfree",
    "DevOps.system.sd2-graphite01.disk._mnt_data.gigabyte_percentfree",
    "DevOps.system.bst-graphite01.disk.root.gigabyte_percentfree",
    "DevOps.system.dtl-graphite01.disk._mnt_data.gigabyte_percentfree"
  ],
  "is_remote": false,
  "mute_new_metrics": false,
  "throttling": 0
}

It is a simple mode trigger, created via API. Simple triggers have only simple thresholds (warn_value and error_value), but no expression. Yet this trigger has four targets. This leads to a non-obvious error: Moira silently ignores all targets for this trigger, except the first one.

We should not allow creating triggers in simple mode with several targets via API. Instead, we should return an error.

Hi,

v2.2.0-1
When using api to create contact with certain ID

curl -s -X PUT http://localhost:80/api/contact -d @moira_objects/contact.json
{"type":"mail","value":"[email protected]"}

the contact is created, but without ID.

contact.json file is:

{
  "type": "mail",
  "value": "[email protected]",
  "id": "4f8bb896-a213-443e-bab7-3105763910ae",
  "user": ""
}

Now tags are always united with "and", it would be cool to have an opportunity to unite tags in subscriptions with logical expressions like "and", "or", "not", etc.
For example:

• tag1 && (tag2 || tag3)
• tag1 && tag2 && !tag3
• (tag1 && tag2) || ( tag1 && tag3)

It's often convenient to include an actionable description for each trigger to improve learning curve for new on-call engineers. However, these descriptions are generally quite long. This causes messages to occupy a lot of screen space and create unnecessary bloat:

There's also a good practice to put discussion regarding each alert into its own thread, which makes the channel easier to navigate:

We propose two improvements in slack notification system to help establish the thread-per-alert practice in on-call channels:

1. An option to automatically start a thread each time a trigger fires and post its description in this thread as a first message. This only applies to harmful transitions (like X --> ERROR).

2. Weave messages about transitions which bring the trigger back into normal state (like X --> OK) into the threads caused by respective harmful transitions.

With these capabilities, one would be able to organize the channel as a stream of "incidents", each with its own history thread.

Заметили странное поведение - алерты от Мойра приходят во интервал времени, который не входит в "watch time".
Т.е. например, в watch time указано "At specific interval: 12:00 - 23:59"
а уведомление в телегу прилетело в 10:09.
Проверьте, пожалуйста.
Спасибо.

Integrate python-moira-client to test API in travis

Hello,

Is there any way to change color and state of metric when turning on the maintenance? For example, if we have error (red) state for some metric but we turn maintenance for it (because we wan't to ignore it for some time), as I understand we won't get any notification for it. But is it possible to change state from error to another?

I'm aware of the python client. Is it the only way to describe and apply triggers programmatically?

Now on moira-test there isn't "expression" key in response on request for trigger data. In dev- and production-moira this key exist. Where is truth?

config notifire

  host: localhost
  port: "6379"
  dbid: 0
graphite:
  enabled: ""
  uri: localhost:2003
  prefix: moira
  interval: 60s0ms
log:
  log_file: stdout
  log_level: debug
notifier:
  sender_timeout: 10s0ms
  resending_timeout: "24:00"
  senders:
    - type: mail
      smtp_host: mail.example.ru
      smtp_port: 25
      mail_from: [email protected]
      insecure_tls: true
  moira_selfstate:
    enabled: "false"
    redis_disconect_delay: 30
    last_metric_received_delay: 60
    last_check_delay: 60
    contacts: []
    notice_interval: 300
  front_uri: http://alert.example.ru
timezone: Europe/Moscow```

add delivery channel and test Email [email protected] 
in log 
Nov  8 11:56:48 moira moira-notifier[32677]: 2017-11-08 11:56:48.411#011notifier#011WARNING#011Can't send message after 3 try: Unknown contact type [package of 1 notifications to [email protected]]. Retry again after 1 min
Nov  8 11:56:48 moira moira-notifier[32677]: 2017-11-08 11:56:48.411#011notifier#011DEBUG#011Scheduled notification for contact email:[email protected] trigger  at 2017/11/08 11:57:48 (1510131468)

We can support HTML-template for emails in external file.

Sometimes something bad happens. And many of your triggers fall into NODATA (for example).
It would be nice if Moira could notice the massive problems and replace one million messages with one. One message with the text "everywhere everything is bad" is much more informative than a flurry of alerts.

При открытии мобильной мойры по ссылке на несуществующий триггер видим вечную крутяшку. Хочется видеть нормальное поведение интерфейса.

not found readme file or any doc :)

file storage-schemas.conf - needed in package moira-checker, moira-cli, moira-filter, moira-notifier?
this is problem with install - error:
attempt to overwrite "/etc/moira/storage-schemas.conf", which is already available in the moira-api 2.0-beta1.2-1 package

Всем привет!
Во всех API интерфейсах графита, есть возможность рендерить картинку относительно метрики.
Очень хочется, иметь возможность, в нотификаторе, включить рассылку картинок для Slack и почты, как это сделано к примеру в Графане.
Картинку можно генерить онлайни и заливать её на расшаренный WEBDAV сервер (для начала). После чего постить ссылку на неё в слак и почту.

in config notifier.yml
timezone: Europe/Moscow
in web interface - all ok
in email notifification - time UTC

I have setted up some alerts and chosen to set OK state if there are no values. Part of my alerts works fine. However I'm getting notifications about NODATA state for those metrics that never had values.

Hello,

Is there any http api for exporting triggers, delivery channels and subscriptions (e.g. to json) to backup them, and any api for their import to new moira installation?

Just got alert email with timestamp - 09:51 17.01.2018.
Is it possible to change it to ISO8601 with time zone?

This test in unstable while build on travis, for example in this build https://travis-ci.org/moira-alert/moira/builds/295220265

Hi! It looks like windows \r\n is not allowed as line delimiter in moira however for example collectd uses this delimiter since the very beginning (see https://github.com/collectd/collectd/blob/master/src/utils_format_graphite.c).
Could you please allow these two characters as separators?

При удалении last check'а мы удаляем все метрики по паттернам триггера. После этого, если по оставшимся каким-то из last check'ов больше не приходят метрики, то у этих last check'ов больше не будет происходить поверка и никогда не поменяется состояние.

Как это сказывается на пользователе:

1. Мы никогда больше не отравим по таким метрикам эвент, о том что метрика в NODATA
2. Мы не удалим такие метрики автоматически, если триггер имеет NODATA state == DEL
3. Мы никогда не уберем последнее значение метрики, если оно не было равно NIL

Как решать проблему:
Если после получения метрик мы понимаем, что у нас есть какие-то last check'и по которым нет метрик, сделать на них checkForNoData

Trying to install moira-cli using
rpm -i moira-cli-2.0.0-1.x86_64.rpm
the command fails with the message

file/usr/lib/systemd/system from install of moira-cli-2.0.0-1.x86_64 conflicts with file from package systemd-219-42.el7_4.4.x86_64

this happens because the package contains the empty directory /usr/lib/systemd/system/

[root@moira02 moira_pkg]# rpm -qlp moira-cli-2.0.0-1.x86_64.rpm
/etc/moira/cli.yml
/usr/bin/moira-cli
/usr/lib/systemd/system

notifier.yml

  senders:
    - type: telegram
      api_token: "xxxxxxxxxxxxxxxxx"

token is work for grafana bot - 100%
in log
Nov 8 17:30:23 moira moira-notifier[8659]: 2017-11-08 17:30:23.243#011notifier#011WARNING#011Can't send message after 12 try: Failed to send message to telegram contact @ihard: failed to get username uuid: Nil returned. . Retry again after 1 min

Разобраться, чем был вызван этот всплеск, который непонятно откуда появился. Кажется на основе данных во сложениях можно спокойно писать тест

Hi

I'm trying to set up Moira.
Install it using released deb packages (from here https://github.com/moira-alert/moira/releases). Version 2.2.0-1

Trying to set notification, but no success.

notifier log shows:
WARNING Can't send message after 41 try: Unknown contact type 'mail' [package of 1 notifications to [email protected]]. Retry again after 1 min

--- notifier.yml

redis:
  host: localhost
  port: "6379"
  dbid: 0
graphite:
  enabled: true
  uri: "localhost:2003"
  prefix: DevOps.Moira
  interval: 60s
log:
  log_file: stdout
  log_level: debug
notifier:
  sender_timeout: 10s
  resending_timeout: "1:00"
  senders:
    - type: mail
      mail_from: moira
      smtp_host: smtp.gmail.com
      smtp_port: 465
      # Skip SMTP server certificate chain validation if false
      insecure_tls: true
      # Uses "mail_from" if empty
      smtp_user: [email protected]
      smtp_pass: pass
      # Email template file path (standard Go templates), if empty use default template
      # template_file: ...
  moira_selfstate:
    enabled: true
    redis_disconect_delay: 60s
    last_metric_received_delay: 120s
    last_check_delay: 120s
    notice_interval: 300s
  front_uri: http://localhost
  timezone: Europe/Moscow

--- web.json

{
  "api_url": "/api",
  "contacts": [
    {"type": "mail", "validation": "^.+@.+\\..+$", "icon": "email"},
    {"type": "pushover", "validation": "", "img": "pushover.ico"},
    {"type": "slack", "validation": "^[@#].+$", "img": "slack.ico"},
    {"type": "telegram", "validation": "^@.+$", "img": "telegram.ico", "title": "@channel only", "help": "required to grant @KonturMoiraBot admin privileges"},
    {"type": "twilio sms", "img": "twilio_sms.ico"},
    {"type": "twilio voice", "img": "twilio_voice.ico"}
  ],
  "supportEmail": "[email protected]"
}

Hello.
I have the notifications with notifier/mail. The trouble is that the text lines have been concatinated in one like:
Description: Heap Size Usage : http://server/dashboard/db/call-manager Please, don’t use mon-alert.link - it is for monitoring admins group purpose.
The < br > or \n line break are not works and transfer as is:
Description: Heap Size Usage :<br> http://server/dashboard/db/call-manager \nPlease, don’t use mon-alert.link - it is for monitoring admins group purpose.
According to source code all the tags are forcibly rewritten.
I need ability of escaping html tags inside Description field in trigger for manual formating own messages.

Хочется в результате поиска выводить теги совпадающие по префиксу, а потом уже все остальные.

Добрый день!

Скажите, а вы не рассматривали использование prometheus (https://prometheus.io) как дополнительный источник метрик?

Trigger Name must be unique. It is necessary for a better automatization.

У триггера сейчас нет метрик из за этого они не переходят в состояние NODATA (Это другой баг). Но пользователю не понятно почему сам триггер в NODATA

ToDo:

• дизайн
• фронтенд
• бэкхенд

We want a service (daemon) to track statistics from different sources (graphite is one of the sources).

In case the statistics indicate problems in our application, the daemon will help to display a message for users on the application page.

Отжирает все коннекции при большой нагрузке

It seems that /api/trigger/<trigger_id>/state
returns the last registered value even if trigger is removed

I can write in expression box mostly whatever I want:

Checkers break down after restart redis

Add getTriggerByName api method. Its needs for ansible module.

The RPMs of moira-filter, moira-checker and moira-notifier install malformed config files; in each there is at row 6:

graphite:
  enabled: ""

instead of

graphite:
  enabled: true

so starting services fails with errors like these:

gen 09 09:21:19 moira-filter[2044]: Can not read settings: Can't parse config file [/etc/moira/filter.yml] [yaml: unmarshal errors:
gen 09 09:21:19 moira-filter[2044]: line 6: cannot unmarshal !!str `` into bool]

/etc/moira/notifier.yml has one more problem:

  moira_selfstate:
    enabled: "false"

instead of

  moira_selfstate:
    enabled: false

so strarting the service returns this error

gen 09 10:34:57 moira-notifier[4574]: line 18: cannot unmarshal !!str false into bool]

We could add different types of triggers

• Push Type for real time alerting. It is like current trigger type.
• Pull Type for simple detect anomalies. This trigger has to pull data from graphite storage periodically, like all other monitoring systems based on graphite. (bosun, beacon)
• HeathCheck Type is a standard shedulled check HTTP, FTP and some other popular protocols like (statuscake.com, uptimerobot.com)
• WatchDog is a trigger that waits for periodical http requests that sets its state. It is necessary for tracking the state of sparse and periodical operations like backups (healthchecks.io, deadmanssnitch.com)

For example:

GET /api/status
{
"status":"fail",
"redis":"ok",
"filter": "ok", 
"checker": "fail"
}

Notifier tries to deliver messages to the contact that has been deleted.
After deleting the subscription - messages are still trying to get.
Is there a ttl?

in log:

2018-01-09 22:18:47.830 notifier        WARNING Can't send message after 106 try: Failed to send message to telegram contact Moira-alerts: failed to get username uuid: Nil returned. . Retry again after 1 min
2018-01-09 22:18:47.830 notifier        WARNING Can't send message after 85 try: Failed to send message to telegram contact Moira-alerts: failed to get username uuid: Nil returned. . Retry again after 1 min

Suddenly, the formatting of our application and bot messages stopped working. About 3-4 weeks ago. At the same time, the formatting works in the mobile application.

I suspect that the issue is the parameter here:

The correct name for this parameter is "mrkdwn". You can find more details here: https://api.slack.com/docs/message-formatting

Hello,
We need alerts only for interval 1-25 jan, 1-25 apr, 1-25 jul, 1-25 oct (да, да, это отчетсность по НДС).
This is because our metrics could show 0 value on days outside these intervals.

Does moira support tags for configure triggers ?
For tags I mean Graphite since 1.1.x series, supports storing data using tags to identify each series. (https://graphite.readthedocs.io/en/latest/tags.html)