fhg-imw / aeternitas Goto Github PK

A pollable which (due to excessive errors) is deactivated should have some kind of Method to help reactivate it again.

Ideally this is also accessible via the WebUI

Provide a API to access Æternitas' metrics

Metrics stored in redis

Grouped by pollable class

• job execution count
• failed job count, failure_ratio
• ignored errors
• deactivations
• execution time
• guard
• waited
• guard wait sleep
• guard timeout exceeded

Data statistics

• number of sources (per pollable class)
• pollables created (per pollable class)
• future pollings (per pollable class)

Using REDIS_URL is per default used by many other redis libraries (i.e. sidekiq, redis-rb).
Aeternitas should respect this behaviour

To avoid DDOSing by accident we should use class level lock key (instead of object level which effectively would not lock in most cases).

We should consider adding a testing mode. This is especially necessary for the sources, as we need to make sure to delete all source files after the source entries are cleaned up.

For the current dummy project I get the following error message when trying to poll:

2017-05-18T08:21:07.798Z 12581 TID-qqp58 Aeternitas::Sidekiq::PollJob JID-9e2adfb1249828ab8fb62110 INFO: start
2017-05-18T08:21:07.824Z 12581 TID-qqp58 Aeternitas::Sidekiq::PollJob JID-9e2adfb1249828ab8fb62110 INFO: fail: 0.026 sec
2017-05-18T08:21:07.824Z 12581 TID-qqp58 WARN: {"context":"Job raised exception","job":{"queue":"polling","class":"Aeternitas::Sidekiq::PollJob","args":[8],"retry":4,"unique":"until_executed","unique_args":[8],"unique_job_expiration":2592000,"jid":"9e2adfb1249828ab8fb62110","created_at":1495092946.0096555,"unique_prefix":"uniquejobs","unique_digest":"uniquejobs:aff129ff0a06f1286ac7d896094cacb3","enqueued_at":1495095667.7981665,"error_message":"invalid uri scheme ''","error_class":"ArgumentError","failed_at":1495095212.355153,"retry_count":1,"retried_at":1495095667.8240654},"jobstr":"{"queue":"polling","class":"Aeternitas::Sidekiq::PollJob","args":[8],"retry":4,"unique":"until_executed","unique_args":[8],"unique_job_expiration":2592000,"jid":"9e2adfb1249828ab8fb62110","created_at":1495092946.0096555,"unique_prefix":"uniquejobs","unique_digest":"uniquejobs:aff129ff0a06f1286ac7d896094cacb3","enqueued_at":1495095667.7981665,"error_message":"invalid uri scheme ''","error_class":"ArgumentError","failed_at":1495095212.355153,"retry_count":0,"retried_at":1495095426.0216072}"}
2017-05-18T08:21:07.824Z 12581 TID-qqp58 WARN: ArgumentError: invalid uri scheme ''
2017-05-18T08:21:07.824Z 12581 TID-qqp58 WARN: /home/max/.rvm/gems/ruby-2.2.6/gems/redis-3.3.3/lib/redis/client.rb:416:in _parse_options' /home/max/.rvm/gems/ruby-2.2.6/gems/redis-3.3.3/lib/redis/client.rb:81:in initialize'
/home/max/.rvm/gems/ruby-2.2.6/gems/redis-3.3.3/lib/redis.rb:51:in new' /home/max/.rvm/gems/ruby-2.2.6/gems/redis-3.3.3/lib/redis.rb:51:in initialize'
/home/max/.rvm/gems/ruby-2.2.6/bundler/gems/aeternitas-9e45aa560e3c/lib/aeternitas.rb:21:in new' /home/max/.rvm/gems/ruby-2.2.6/bundler/gems/aeternitas-9e45aa560e3c/lib/aeternitas.rb:21:in block in redis'
/home/max/.rvm/gems/ruby-2.2.6/gems/connection_pool-2.2.1/lib/connection_pool/timed_stack.rb:170:in `call'

An application may want to explicitly check wether a guard is locked/sleeping etc. (for instance when testing that a rate limit exception causes the resource to be locked accordingly).

Currently guard does not offer this information via public API. There exist private methods (get_timeout, get_payload) which may be used but a dedicated method would be better.

Add an extensive usage documentation

Implement a rails engine with good usability to utilize the metrics collected via #2