Checks not running about cabot HOT 11 CLOSED

Is there anything in your logs (fab logs -H your.host)?

This is weird because if you've run a check in the last 10 mins it should show up "Checks running" - if you manually run the checks and then immediately ping the status endpoint do you still see "Checks not running"?

from cabot.

Have run a service update (2 checks) just now and the status did not change.

Here are the logs:

[host] Executing task 'logs'
[host] sudo: tail -f /var/log/cabot/* /var/log/nginx/*.log
[host] out: ==> /var/log/cabot/celery-1.log <==
[host] out:
[host] out: [2014-01-16 02:14:15,728: ERROR/Beat] beat: Connection error: invalid password. Trying again in 32.0 seconds...
[host] out: [2014-01-16 02:14:47,680: ERROR/MainProcess] consumer: Cannot connect to redis://@localhost:6379/1: invalid password.
[host] out: Trying again in 32.00 seconds...
[host] out:
[host] out: [2014-01-16 02:14:47,764: ERROR/Beat] beat: Connection error: invalid password. Trying again in 32.0 seconds...
[host] out: [2014-01-16 02:15:19,716: ERROR/MainProcess] consumer: Cannot connect to redis://@localhost:6379/1: invalid password.
[host] out: Trying again in 32.00 seconds...
[host] out:
[host] out: [2014-01-16 02:15:19,800: ERROR/Beat] beat: Connection error: invalid password. Trying again in 32.0 seconds...
[host] out:
[host] out: ==> /var/log/cabot/web-1.log <==
[host] out: 2014-01-15 19:07:35 [885] [INFO] Booting worker with pid: 885
[host] out: 2014-01-15 19:08:38 [885] [INFO] Worker exiting (pid: 885)
[host] out: 2014-01-15 19:08:38 [884] [INFO] Worker exiting (pid: 884)
[host] out: 2014-01-15 19:08:38 [792] [INFO] Handling signal: term
[host] out: 2014-01-15 19:08:38 [792] [INFO] Shutting down: Master
[host] out: 2014-01-15 19:08:56 [760] [INFO] Starting gunicorn 18.0
[host] out: 2014-01-15 19:08:56 [760] [INFO] Listening at: http://0.0.0.0:5000 (760)
[host] out: 2014-01-15 19:08:56 [760] [INFO] Using worker: sync
[host] out: 2014-01-15 19:08:56 [877] [INFO] Booting worker with pid: 877
[host] out: 2014-01-15 19:08:56 [878] [INFO] Booting worker with pid: 878
[host] out:
[host] out: ==> /var/log/nginx/access.log <==
[host] out: 100.100.100.100 - - [15/Jan/2014:21:11:21 -0500] "GET /static/CACHE/js/d5d180aba8b8.js HTTP/1.1" 304 0 "http://host/services/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
[host] out: 100.100.100.100 - - [15/Jan/2014:21:11:22 -0500] "GET /static/bootstrap/fonts/glyphicons-halflings-regular.woff?4dbad1e98ae9 HTTP/1.1" 304 0 "http://host/services/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
[host] out: 100.100.100.100 - - [15/Jan/2014:21:11:25 -0500] "GET /service/run_service_update/1/ HTTP/1.1" 302 5 "http://host/services/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
[host] out: 100.100.100.100 - - [15/Jan/2014:21:11:25 -0500] "GET /service/1/ HTTP/1.1" 200 2899 "http://host/services/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
[host] out: 100.100.100.100 - - [15/Jan/2014:21:11:26 -0500] "GET /static/CACHE/js/f4f966db6286.js HTTP/1.1" 304 0 "http://host/service/1/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
[host] out: 100.100.100.100 - - [15/Jan/2014:21:11:37 -0500] "GET /status/ HTTP/1.1" 200 49 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
[host] out: 100.100.100.100 - - [15/Jan/2014:21:11:38 -0500] "GET /favicon.ico HTTP/1.1" 404 1304 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
[host] out: 100.100.100.100 - - [15/Jan/2014:21:11:40 -0500] "GET /status/ HTTP/1.1" 200 49 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
[host] out: 100.100.100.100 - - [15/Jan/2014:21:11:41 -0500] "GET /favicon.ico HTTP/1.1" 404 1304 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
[host] out: 100.100.100.100 - - [15/Jan/2014:21:11:57 -0500] "-" 400 0 "-" "-"
[host] out:
[host] out: ==> /var/log/nginx/error.log <==
[host] out:
[host] out: ==> /var/log/cabot/celery-1.log <==
[host] out: [2014-01-16 02:15:51,752: ERROR/MainProcess] consumer: Cannot connect to redis://@localhost:6379/1: invalid password.
[host] out: Trying again in 32.00 seconds...
[host] out:
[host] out: [2014-01-16 02:15:51,835: ERROR/Beat] beat: Connection error: invalid password. Trying again in 32.0 seconds...
[host] out: [2014-01-16 02:16:23,787: ERROR/MainProcess] consumer: Cannot connect to redis://@localhost:6379/1: invalid password.
[host] out: Trying again in 32.00 seconds...
[host] out:
[host] out: [2014-01-16 02:16:23,870: ERROR/Beat] beat: Connection error: invalid password. Trying again in 32.0 seconds...
[host] out:
[host] out: ==> /var/log/nginx/access.log <==
[host] out: 100.100.100.100 - - [15/Jan/2014:21:16:28 -0500] "GET /status/ HTTP/1.1" 200 49 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
[host] out: 100.100.100.100 - - [15/Jan/2014:21:16:35 -0500] "GET /service/run_service_update/1/ HTTP/1.1" 302 5 "http://host/service/1/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
[host] out: 100.100.100.100 - - [15/Jan/2014:21:16:36 -0500] "GET /service/1/ HTTP/1.1" 200 2909 "http://host/service/1/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

from cabot.

It looks like I used the wrong name for SES_USER. I updated it and ran fab -H deploy host and /status/ now reports checks are running.

Fixing the SES user also has the HipChat and SMS notifications being sent.

from cabot.

@anthonysexton glad you resolved, although looking at the logs above it seems that one problem is/may be your redis connection. Is redis installed correctly and is the CELERY_BROKER_URL setting correct?

from cabot.

@dbuxton The deploy was done using the fabric scripts. Was there anything additional I was meant to do to sort out Redis?

On Thursday, 16 January 2014 at 21:32, dbuxton wrote:

@anthonysexton (https://github.com/anthonysexton) glad you resolved, although looking at the logs above it seems that one problem is/may be your redis connection. Is redis installed correctly and is the CELERY_BROKER_URL setting correct?

—
Reply to this email directly or view it on GitHub (#7 (comment)).

from cabot.

Well, the password you set in conf/production.env (see https://github.com/arachnys/cabot/blob/master/conf/production.env.example#L17) should match the one set in bin/setup_dependencies.sh (https://github.com/arachnys/cabot/blob/master/bin/setup_dependencies.sh#L82) - to be honest it's not as simple/joined up as it should be as we don't use this provisioning script in-house and I just put it together so people could get started quickly.

You can change the password required on the server by editing the /etc/redis/redis.conf on the server directly - running setup_dependencies.sh again won't change a password that's already set. Then change the broker url to match.

from cabot.

Why not disable redis password for default provisioning? I've been bit by same issue (celery not working after setup).

from cabot.

Probably a good idea...

from cabot.

Strange. I just had this error after changing some settings and doing redeploy. Of course I haven't touched redis settings in the config.

To resolve this I had to edit redis password set in this config variable https://github.com/arachnys/cabot/blob/master/conf/production.env.example#L22 (not like 17 anymore) to the one set in https://github.com/arachnys/cabot/blob/master/bin/setup_dependencies.sh#L82 and do the redeploy.

from cabot.

I think it's because redis has been upgraded on the redeploy, version isn't locked down...

from cabot.

FYI, I had to change redis password from "yourrediskey" to "yourredispassword" in conf/production.env in my Ubuntu 12.04.5 deployment, and then cabot/celery begins working and checking status.

from cabot.