I'm wondering about the recommended way to monitor basic host metrics (reported by collectd, for example, "load").

My first though was just to use wildcards, to monitor the metric for all hosts:

collectd.*.load.load.shortterm

The problem with that is that you have to drill into the check to find out which host is actually misbehaving - it doesn't show up in the email, the service page, the checks page or the check results overview - I have to go to the check result detail and use the graph hover pop-up, which can be hard to read if the metric name is long.

I also thought about creating a service for each host, but I can't find an easy way to duplicate services or checks. It seems like a lot of work to recreate a list of checks for each host.

I would love to see something like service templates, where you could insert a variable to be used in the metric checks. Then you could create a service, fill out the variable name (which could be the hostname, or something else) and you be all set.

Am I missing something? Thanks!

Please add the ability to send a test sms/email/phone-call etc. so that I can verify the alerting channels work

This is both a report for a minor immediate problem, and also a request for more general embedding of the recovery documentation.

The immediate issue is that the Hackpad subdomain fro Arachnys seems to be hardcoded here:

I initially supposed that the fix could be adding a configured env variable to determine the Hackpad subdomain. However, I think there is a much more flexible solution which may not be much more difficult to implement. My suggestion is to support configuring HTML snippets instead of a Hackpad ID as the source of the instructions. That would instantly enable other subdomains of Hackpad, as well as any other embeddable documents. For example, Gists.

Thanks for building and sharing an awesome tool!

Currently we're updating Service status every minute. That's OK but it means there's a lag between a check failing (or recovering) and the service being updated. This, combined with varying polling frequencies of the checks themselves, can make it hard to work out what's going on if things are bouncing around.

The easiest fix here is to trigger service recalculation after every check run. Originally we didn't do this because we were concerned that it would trigger too many alerts but now alerts and notifications are throttled to one every ten minutes it doesn't matter how frequently we update.

Firstly, i'm using sqlite database. Trying to Text Match an http check produces this when i run the check.
Also when check crashes it's not marked as failed of warning, so basically check can fail silently in the background you'd never know.

[2014-01-26 14:01:58,477: ERROR/MainProcess] Task app.cabotapp.tasks.run_status_check[20005a7b-ee17-4d85-8cfc-9824d02df62f] raised unexpected: DatabaseError('You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.',)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 238, in trace_task
R = retval = fun(_args, *_kwargs)
File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 416, in protected_call
return self.run(_args, *_kwargs)
File "/cabot/app/cabotapp/tasks.py", line 42, in run_status_check
check.run()
File "/cabot/app/cabotapp/models.py", line 503, in run
result.save()
File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 463, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 551, in save_base
result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 203, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1593, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 912, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/util.py", line 40, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/sqlite3/base.py", line 344, in execute
return Database.Cursor.execute(self, query, params)
DatabaseError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

Provisioning script doesn't run on newer versions of vagrant (mine is 1.4.0) and the vagrantfile formant looks a little out of date.

I set up cabot and put my correct SES user information in the production config file. However, the app does not send any emails when a check fails. I checked both the SNS log and my mailserver log - the email is never sent.
I also made sure ever checkbox saying e.g. "enable notification" is checked.

I would like to configure the value of the filter of time on graphite of every check, because now is fixed for all the system with "-10min" and i have some graphs that i need more time to decide if is correct or not the check

For example:

BEGIN:VEVENT
DTSTART;TZID=America/Mexico_City:20140305T100000
DTEND;TZID=America/Mexico_City:20140305T150000
RRULE:FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR
DTSTAMP:20140313T231346Z
UID:[email protected]
CREATED:20140306T042702Z
DESCRIPTION:
LAST-MODIFIED:20140313T231344Z
LOCATION:
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:ipmb
TRANSP:OPAQUE
END:VEVENT

I think it would be great if cabot can get alerts from sensu.
Thanks.

Will you accept pull request fixing E111 indentation is not a multiple of four? I found no code guideline for the project http://cabotapp.com/dev/contribute-code.html

I think for OSS project, PEP8 compliance is a good thing (practically, for readabilty).
If you're ok with proposed change I'll submit corresponding fixes.

Trying to provision a Cabot instance on AWS, I get a permission denied error.

Steps:

• Launch AWS instance using Ubuntu 13.04
• Run Fabric provision task

I run:
$ fab provision -i ~/.ssh/my-aws-keypair.pem -H [email protected] -u ubuntu

And get result:

[[email protected]] Executing task 'provision'
[[email protected]] put: bin/setup_dependencies.sh -> /tmp/setup_dependencies.sh
[[email protected]] run: LOCAL_SSH_KEY="ssh-rsa [REDACTED] [email protected]" bash /tmp/setup_dependencies.sh
[[email protected]] out: /tmp/setup_dependencies.sh: line 15: /etc/sudoers.d/ubuntu: Permission denied
[[email protected]] out:


Fatal error: run() received nonzero return code 1 while executing!

Requested: LOCAL_SSH_KEY="ssh-rsa [REDACTED] [email protected]" bash /tmp/setup_dependencies.sh
Executed: /bin/bash -l -c "LOCAL_SSH_KEY=\"ssh-rsa [REDACTED] [email protected]\" bash /tmp/setup_dependencies.sh"

Aborting.
Disconnecting from ec2-123-123-123-123.compute-1.amazonaws.com... done.

I have an issue where no background checks are performed. The logs are not showing anything unusual. What can I check next?

Hello,

I am getting 404 errors when running the server. It seems like the cache is not generated.

GET http://-----/static/CACHE/js/490dff075d3d.js 404 (NOT FOUND) 

Here are the logs:

2b91824cd7c8# foreman run -f=Procfile -e=conf/production.env python manage.py runserver 0.0.0.0:8000
Validating models...

0 errors found
Django version 1.4.10, using settings 'app.settings'
Development server is running at http://0.0.0.0:8000/
Quit the server with CONTROL-C.
[04/Mar/2014 18:07:18] "GET /accounts/login/?next=/services/ HTTP/1.1" 200 4591
[04/Mar/2014 18:07:18] "GET /static/CACHE/css/9e0d6607d8b0.css HTTP/1.1" 404 3390
[04/Mar/2014 18:07:18] "GET /static/CACHE/js/27b97496ae85.js HTTP/1.1" 404 3390
[04/Mar/2014 18:07:19] "GET /static/CACHE/js/490dff075d3d.js HTTP/1.1" 404 3390
[04/Mar/2014 18:07:19] "GET /static/arachnys/img/icon_48x48.png HTTP/1.1" 404 3390

Any ideas?

I wrote an installation manual for developers using OS X to get start without running a virtual server (which is quite overkill for this). If you like it, please include it in your documentation.

https://gist.github.com/jirutka/8636572

Haven't checked what's wrong yet.

ERROR 2014-01-24 12:57:03,371 base 8628 140590461187840 Internal Server Error: /accounts/password-reset/
Traceback (most recent call last):
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/utils/decorators.py", line 91, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/contrib/auth/views.py", line 153, in password_reset
    form.save(**opts)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/contrib/auth/forms.py", line 241, in save
    email = loader.render_to_string(email_template_name, c)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/template/loader.py", line 171, in render_to_string
    return t.render(Context(dictionary))
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/template/base.py", line 140, in render
    return self._render(context)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/template/base.py", line 134, in _render
    return self.nodelist.render(context)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/template/base.py", line 823, in render
    bit = self.render_node(node, context)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/template/base.py", line 837, in render_node
    return node.render(context)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/template/defaulttags.py", line 31, in render
    output = self.nodelist.render(context)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/template/base.py", line 823, in render
    bit = self.render_node(node, context)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/template/base.py", line 837, in render_node
    return node.render(context)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/template/loader_tags.py", line 53, in render
    result = self.nodelist.render(context)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/template/base.py", line 823, in render
    bit = self.render_node(node, context)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/template/base.py", line 837, in render_node
    return node.render(context)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/template/defaulttags.py", line 426, in render
    raise e
NoReverseMatch: Reverse for 'django.contrib.auth.views.password_reset_confirm' with arguments '()' and keyword arguments '{'uidb36': '3', 'token': '**********'}' not found.

Currently if a key does not exist, the default appears to be an empty array. Instead this should result in the check failing.

Hi,

I cloned and deployed cabot using fab, it went smoothly. Opened the required port on EC2 security management, changed web app host name in production.env to match the installation.

However, when I open it in the browser, assets are not loading and it shows 404 errors for them in the dev tools. URLs look current ( http://somewhere.amazonaws.com:5000/static/arachnys/img/icon_48x48.png etc ), the requested files exist in build_dir/static/, and they have read for all permission. So can't figure out what might be an issue.

Is there something else I should try to fix it?

It's running on EC2 Ubuntu, ubuntu user existed already (since it's the default one).

Thanks!

I've just updated m Cabot instance by doing a git pull and re-running fab deploy. However if I save or run the check I've disabled SSL verification on, I see "Unhandled error. Please look at logs."

The logs seem similar to issue #7 , in that I'm seeing Redis connection errors. Looking at that issue, you said to make sure the password on line 17 of the production.conf file was the same as in redis.conf, but that doesn't look like a password? I've updated them to be the same (they aren't in Git), and the error changes to "operation not permitted". I think this is because the special characters I have in my password are messing with the Redis connection URL.

I may be doing several things wrong here - happy to step back a bit and see if we can figure it out.

Background

We have been using cabot to monitoring our company's projects for couple of months by using http checks, it works pretty well.

But few days ago, we found that it stopped to do regular auto-checks, and even it was failed in a manual check, it did not send alert messages to subscribers.

We have fixed this issue by updating to latest code (today), but still no ideas what caused the previous problems.

Below is one of the errors we found in the /var/log/cabot/celery-1.log, besides that no error messages found in /var/log/cabot/celery-1.log nor /var/log/web-1.log .

These are the only information we can delivery to trace the problems.

Error logs

[2014-02-18 09:40:12,513: ERROR/MainProcess] Task app.cabotapp.tasks.update_shifts[b690a241-39b6-49f6-8764-92b119f8ff3e] raised unexpected: ValueE
rror('Content line could not be parsed into parts: \'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"><ht
ml><head><meta http-equiv="content-type" content="text/html;charset=utf-8">\': <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.or
g/TR/html4/strict.dtd"><html><head><meta http-equiv="content-type" content="text/html;charset=utf-8">',)
Traceback (most recent call last):
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 238, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 416, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/ubuntu/2014-01-17-ea7deb1/app/cabotapp/tasks.py", line 80, in update_shifts
    _update_shifts()
  File "/home/ubuntu/2014-01-17-ea7deb1/app/cabotapp/models.py", line 668, in update_shifts
    events = get_events()
  File "/home/ubuntu/2014-01-17-ea7deb1/app/cabotapp/calendar.py", line 16, in get_events
    for component in get_calendar_data().walk():
  File "/home/ubuntu/2014-01-17-ea7deb1/app/cabotapp/calendar.py", line 10, in get_calendar_data
    cal = Calendar.from_ical(resp.content)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/icalendar/cal.py", line 412, in from_ical
    name, params, vals = line.parts()
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/icalendar/parser.py", line 526, in parts
    raise ValueError, "Content line could not be parsed into parts: %r: %s" % (self, e)
ValueError: Content line could not be parsed into parts: '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd
"><html><head><meta http-equiv="content-type" content="text/html;charset=utf-8">': <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w
3.org/TR/html4/strict.dtd"><html><head><meta http-equiv="content-type" content="text/html;charset=utf-8">
[2014-02-18 09:10:12,539: ERROR/MainProcess] Task app.cabotapp.tasks.update_shifts[55c3d9ed-ebda-41c1-adb5-ae0f16e3ac40] raised unexpected: ValueE
rror('Content line could not be parsed into parts: \'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"><ht
ml><head><meta http-equiv="content-type" content="text/html;charset=utf-8">\': <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.or
g/TR/html4/strict.dtd"><html><head><meta http-equiv="content-type" content="text/html;charset=utf-8">',)
Traceback (most recent call last):
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 238, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 416, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/ubuntu/2014-01-17-ea7deb1/app/cabotapp/tasks.py", line 80, in update_shifts
    _update_shifts()
  File "/home/ubuntu/2014-01-17-ea7deb1/app/cabotapp/models.py", line 668, in update_shifts
    events = get_events()
  File "/home/ubuntu/2014-01-17-ea7deb1/app/cabotapp/calendar.py", line 16, in get_events
    for component in get_calendar_data().walk():
  File "/home/ubuntu/2014-01-17-ea7deb1/app/cabotapp/calendar.py", line 10, in get_calendar_data
    cal = Calendar.from_ical(resp.content)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/icalendar/cal.py", line 412, in from_ical
    name, params, vals = line.parts()
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/icalendar/parser.py", line 526, in parts
    raise ValueError, "Content line could not be parsed into parts: %r: %s" % (self, e)
ValueError: Content line could not be parsed into parts: '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd
"><html><head><meta http-equiv="content-type" content="text/html;charset=utf-8">': <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w
3.org/TR/html4/strict.dtd"><html><head><meta http-equiv="content-type" content="text/html;charset=utf-8">
[2014-02-18 08:51:45,650: ERROR/MainProcess] Error on stopping Pool: AssertionError('can only join a started process',)
Traceback (most recent call last):
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/bootsteps.py", line 155, in send_all
    fun(parent, *args)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/bootsteps.py", line 377, in stop
    return self.obj.stop()
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/concurrency/base.py", line 119, in stop
    self.on_stop()
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/concurrency/prefork.py", line 140, in on_stop
    self._pool.join()
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/billiard/pool.py", line 1523, in join
    stop_if_not_current(self._result_handler)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/billiard/pool.py", line 148, in stop_if_not_current
    thread.stop(timeout)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/billiard/pool.py", line 493, in stop
    self.on_stop_not_started()
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 301, in on_stop_not_started
    join_exited_workers(shutdown=True)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/billiard/pool.py", line 1099, in _join_exited_workers
    worker.join()
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/billiard/process.py", line 152, in join
    assert self._popen is not None, 'can only join a started process'
AssertionError: can only join a started process

[2014-02-18 08:40:12,508: ERROR/MainProcess] Task app.cabotapp.tasks.update_shifts[a910d857-8346-4ae4-a4f5-e1818b20a776] raised unexpected: ValueE
rror('Content line could not be parsed into parts: \'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"><ht
ml><head><meta http-equiv="content-type" content="text/html;charset=utf-8">\': <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.or
g/TR/html4/strict.dtd"><html><head><meta http-equiv="content-type" content="text/html;charset=utf-8">',)
Traceback (most recent call last):
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 238, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 416, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/ubuntu/2014-01-17-ea7deb1/app/cabotapp/tasks.py", line 80, in update_shifts
    _update_shifts()
  File "/home/ubuntu/2014-01-17-ea7deb1/app/cabotapp/models.py", line 668, in update_shifts
    events = get_events()
  File "/home/ubuntu/2014-01-17-ea7deb1/app/cabotapp/calendar.py", line 16, in get_events
    for component in get_calendar_data().walk():
  File "/home/ubuntu/2014-01-17-ea7deb1/app/cabotapp/calendar.py", line 10, in get_calendar_data
    cal = Calendar.from_ical(resp.content)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/icalendar/cal.py", line 412, in from_ical
    name, params, vals = line.parts()
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/icalendar/parser.py", line 526, in parts
    raise ValueError, "Content line could not be parsed into parts: %r: %s" % (self, e)
ValueError: Content line could not be parsed into parts: '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd
"><html><head><meta http-equiv="content-type" content="text/html;charset=utf-8">': <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w
3.org/TR/html4/strict.dtd"><html><head><meta http-equiv="content-type" content="text/html;charset=utf-8">
[2014-02-18 08:37:45,543: ERROR/MainProcess] Task app.cabotapp.tasks.update_service[6ae91546-90b2-481a-9330-80daf784481e] raised unexpected: Worke
rLostError('Worker exited prematurely: signal 9 (SIGKILL).',)
Traceback (most recent call last):
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/billiard/pool.py", line 1167, in mark_as_worker_lost
    human_status(exitcode)),
WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL).

After pulling latest master in and restarting my celery workers, I get this err

Error in performing check: 'instancemethod' object is not iterable

it seems to happen for all graphite checks, all HTTP checks continue to work just fine

PostgreSQL is being installed as part of the dependencies by your setup fabric. I was wondering if there is any PostgreSQL specific feature you're using, or I can use MySQL just the same?

I've setup a couple of users, all of whom have a name, email address, phone number and HipChat alias defined.

If I select Fallback alert user for 1 and an alert triggers, nothing appears to send. If I uncheck that selection and manually run an alert, the alert successfully triggers.

Logs are below.

[ubuntu@host] out: [2014-01-17 00:56:42,059: ERROR/MainProcess] Task app.cabotapp.tasks.update_service[3c4384e2-b8a0-4257-8b6f-c09b10fe90d7] raised unexpected: AttributeError("'UserProfile' object has no attribute 'email'",)
[ubuntu@host] out: Traceback (most recent call last):
[ubuntu@host] out: File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 238, in trace_task
[ubuntu@host] out: R = retval = fun(_args, *_kwargs)
[ubuntu@host] out: File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 416, in protected_call
[ubuntu@host] out: return self.run(_args, *_kwargs)
[ubuntu@host] out: File "/home/ubuntu/2014-01-17-b2c22da/app/cabotapp/tasks.py", line 74, in update_service
[ubuntu@host] out: service.update_status()
[ubuntu@host] out: File "/home/ubuntu/2014-01-17-b2c22da/app/cabotapp/models.py", line 142, in update_status
[ubuntu@host] out: self.alert()
[ubuntu@host] out: File "/home/ubuntu/2014-01-17-b2c22da/app/cabotapp/models.py", line 182, in alert
[ubuntu@host] out: send_alert(self, duty_officers=get_duty_officers())
[ubuntu@host] out: File "/home/ubuntu/2014-01-17-b2c22da/app/cabotapp/alert.py", line 36, in send_alert
[ubuntu@host] out: send_email_alert(service, users, duty_officers)
[ubuntu@host] out: File "/home/ubuntu/2014-01-17-b2c22da/app/cabotapp/alert.py", line 55, in send_email_alert
[ubuntu@host] out: emails += [u.email for u in duty_officers]
[ubuntu@host] out: AttributeError: 'UserProfile' object has no attribute 'email'

Dokku uses Docker to create containers very quickly. Since the port is variable, how can I change the configuration to use the env variable of PORT?

I have an HTTP check set up which I'm expecting to return a 200 OK from the login page of my continuous integration server. However, I receive a 401 instead. I am able to reproduce this only by setting the "Authorization" header to "Basic :" which I think would be the value if an empty username and password were concatenated and sent?

I have cabot deployed and a service with 2 HTTP checks setup.

If I trigger the checks to run via the dashboard, they complete successfully however visiting /status/ displays "Checks not running".

Is there something additional I need to do to start the scheduled checks?

I don't see any way to do that. Profile page link would be nice.

Could be awesome to add a simple ping check to see if hosts are up.
Keep up the good work!

I get a lot of AttributeErrors

Exception AttributeError: AttributeError("'_DummyThread' object has no attribute '_Thread__block'",)
in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored

Can it safely be ignored?

When send_mail throws an exception (because, for example, the server may be misconfigured) no other alerts are being sent (SMS etc)

FYI, I'm using https://github.com/ryanaudiencepi/heroku-cabot to run Cabot on Heroku. Is there a limit on how many entries of historical check data that's kept around? I'm running a couple of checks every 10 minutes and I would like to limit this to, say, the last day or some number of fixed entries. Any ideas on how best to do this, if it's not already supported?

It's good idea to check speed of loading with cabot. 1 second is too much - more flexible scale is required here such as 100 ms and like that.
Since requests support non-integral timeouts all that should be done is migration of timeout to DecimalField and new 'help_text'.

Hey guys,
Have you seen this behavior before? Shouldn't the second check be unsuccessful as well?

Warning: 13511 - Say: Invalid voice value

The value of the voice attribute must be "man" or "woman". See the Say Verb API Reference for more information.

If http check is supplied with wrong url, say http:/example.com with one slash,
task fails with uncaught exception.

[2014-01-22 16:23:29,156: ERROR/MainProcess] Task app.cabotapp.tasks.run_status_check[190292be-0630-4781-9700-921ba6ae3ad0] raised unexpected: AttributeError("'NoneType' object has no attribute 'encode'",)
Traceback (most recent call last):
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 238, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 416, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/ubuntu/2014-01-18-65148c3/app/cabotapp/tasks.py", line 42, in run_status_check
    check.run()
  File "/home/ubuntu/2014-01-18-65148c3/app/cabotapp/models.py", line 466, in run
    auth=auth
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/requests/api.py", line 65, in get
    return request('get', url, **kwargs)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/requests/safe_mode.py", line 39, in wrapped
    return function(method, url, **kwargs)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/requests/api.py", line 51, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 241, in request
    r.send(prefetch=prefetch)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/requests/models.py", line 629, in send
    timeout=self.timeout,
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 415, in urlopen
    body=body, headers=headers)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 267, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 958, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 986, in _send_request
    self.putrequest(method, url, **skips)
  File "/usr/lib/python2.7/httplib.py", line 896, in putrequest
    host_enc = self.host.encode("ascii")
AttributeError: 'NoneType' object has no attribute 'encode'

Looks like subpages on the Github.io site for this project are failing. For example

Does cabot only supports Ubuntu?

It's not clear form the docs, that changing conf/production.env won't affect cabot, even after cabot stop/start.

Docs should probably state the fact it's required to make fab deploy (which will trigger setup_upstart, which will update env variables, avaiblable to web and queue servers).

Or maybe it's possible to make production.env effective without upstart update?

Cabot fails the HTTP check upon encountering my internal services with self-signed certificates. I can't seem to find a way to instruct Cabot to accept this?

Hello!

I'm trying to configure cabot to send email notifications, but i can't.
Can i use my local SMTPrelay to send email, or i have to use Amazon SES only?

What the format of configuration strings? Can you please post an example of it.

I have this config, and this error in log file:
SES_HOST=192.168.2.8
SES_USER=cabot
SES_PASS=cabot
SES_PORT=25

[2014-05-07 16:28:07,536: ERROR/MainProcess] Task app.cabotapp.tasks.update_service[81f6691e-caf6-4385-960e-b226be1d6de9] raised unexpected: SSLError(1, '_ssl.c:504: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol')
Traceback (most recent call last):
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 238, in trace_task
R = retval = fun(_args, *_kwargs)
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 416, in protected_call
return self.run(_args, *_kwargs)
File "/home/ubuntu/2014-05-06-5ed4639/app/cabotapp/tasks.py", line 72, in update_service
service.update_status()
File "/home/ubuntu/2014-05-06-5ed4639/app/cabotapp/models.py", line 150, in update_status
self.alert()
File "/home/ubuntu/2014-05-06-5ed4639/app/cabotapp/models.py", line 191, in alert
send_alert(self, duty_officers=get_duty_officers())
File "/home/ubuntu/2014-05-06-5ed4639/app/cabotapp/alert.py", line 36, in send_alert
send_email_alert(service, users, duty_officers)
File "/home/ubuntu/2014-05-06-5ed4639/app/cabotapp/alert.py", line 66, in send_email_alert
recipient_list=emails,
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/core/mail/init.py", line 61, in send_mail
connection=connection).send()
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/core/mail/message.py", line 248, in send
return self.get_connection(fail_silently).send_messages([self])
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django/core/mail/backends/smtp.py", line 85, in send_messages
new_conn_created = self.open()
File "/home/ubuntu/venv/local/lib/python2.7/site-packages/django_smtp_ssl.py", line 12, in open
local_hostname=DNS_NAME.get_fqdn())
File "/usr/lib/python2.7/smtplib.py", line 776, in init
SMTP.init(self, host, port, local_hostname, timeout)
File "/usr/lib/python2.7/smtplib.py", line 249, in init
(code, msg) = self.connect(host, port)
File "/usr/lib/python2.7/smtplib.py", line 309, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/lib/python2.7/smtplib.py", line 782, in _get_socket
new_socket = ssl.wrap_socket(new_socket, self.keyfile, self.certfile)
File "/usr/lib/python2.7/ssl.py", line 381, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python2.7/ssl.py", line 143, in init
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 305, in do_handshake
self._sslobj.do_handshake()
SSLError: [Errno 1] _ssl.c:504: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

Currently it is not possible to configure gmail as your smtp provider.

I had manually to change the settings.py in order to allow emails beings sent throught gmail.

EMAIL_USE_TLS = True
EMAIL_HOST = 'smtp.gmail.com'
EMAIL_PORT = 587
EMAIL_HOST_USER = '[email protected]'
EMAIL_HOST_PASSWORD = 'password'

I had also to remove the line EMAIL_BACKEND from the settings file.

As discussed in #41 cabot should be released on PyPI.

Instead of changing EMAIL_BACKEND to django.core.mail.backends.smtp.EmailBackend by hand to be able to use a non ssl smtp server, it would be nice to have on option in the config file/env (or maybe guess it with the port number ?)

More and more folks will want a wider variety of alerts. The existing built-in alert channels are likely the most commonly desired, but there are probably many other less-used channels that users would like to use (in our case, it's slack, and I see other specific suggestions in the issues).

For our use case, and many similar cases, we could go through a generic shell alert. We would configure a shell snippet, and perhaps relevant cabot variables (e.g. names of check and service that are failing) could be passed in as environment variables. For our case, and likely most others, a curl 1-liner is all we need.

This would give users a ton of flexibility. Cabot would immediately be able to configurably alert on vastly more possibly channels.

(Another similar path could be a generic HTTP hook channel, where users can construct HTTP requests as well as endpoints. However that would be probably be more work and would serve fewer use cases.)

Stashboard is a python status page with a rest api.
http://www.stashboard.org/

It would be great if cabot could update the status page automatically.

Does cabot support centos?

Any references for installing cabot on centos?

Didn't expect http checks to follow redirects, but they do. I don't see any option to disable this. I think by default check should never follow http redirects, at least that's what i'm used to.

Hi there, I'm doing a trial on hosted graphite and was curious if and how you integrate with their access keys. Your documentation suggested Base64 auth with Graphite user/pass. Is it simply a matter of using their access key as the username with no password?

I am having issues in installing cabot on an AWS / small instance. I am getting this error in the end of the deployment process.

[ec2-54-86-233-243.compute-1.amazonaws.com] out: Cleaning up...

[ec2-54-86-233-243.compute-1.amazonaws.com] sudo: foreman run python manage.py syncdb -e conf/production.env
[ec2-54-86-233-243.compute-1.amazonaws.com] out: Unknown command: 'syncdb'
[ec2-54-86-233-243.compute-1.amazonaws.com] out: Type 'manage.py help' for usage.