Context

To begin with, Goose chose Carmine because it was popular, stable, well-maintained & most importantly, it solved basic needs of pushing to & popping from a list.

Issues

With passage of time, we've discovered certain issues:

1. Carmine doesn't support closing connections cleanly. More details can be found in Carmine's issue #266 and issue #224
2. Carmine doesn't support commands introduced in Rredis 6.2.0.LMOVE command is needed to enqueue in-progress jobs to the front of Job queue. Lua scripting or atomic transactions are difficult to implement for this task.

As a workaround, Goose sends dummy value to a utility queue. This helps exit the blocking call as it receives a message it was waiting on.

Requirements

Ideally, the client should help Goose handle it's connection-pool, be well-maintained, support redis cluster, etc.

Options

1. Celtuce and Obiwan have provisions for closing a connection.
   Cons: They don't seem well-maintained & stable. When spiking them locally, I observed random issues like thread not closing, unexpected serializations, etc.
2. Write a simple wrapper around Jedis or use any stable Java library using Interop

For Redis, RabbitMQ & Amazon SQS, sketch out approach, Interface & feasible features.

Issue

Goose has many context-heavy keywords & the same keyword might mean multiple things depending on time & perspective of object.

Create a Glossary defining the following:

• broker
• threads
• queues
  • queue
  • schedule-queue
  • retry-queue
  • dead-queue
  • in-progress-queue, preservation-queue & orphan-queue
• ...

• Use an in-progress queue for abrupt worker process shutdowns
• Use LMOVE right left, or BRPOPLPUSH as redis 6.2 is relatively new
• Checkout https://klotzandrew.com/blog/sidekiq-lost-messages
• Goose has a single queue per worker because reliability cannot be achieved using multiple queues in redis as mentioned here

In the beginning a Job can be:

• enqueued
• scheduled
• cron-scheduled

As part of it's lifecycle, the job could be:

• enqueued to ready-queue by a scheduler
• fail execution
• be retried
• be dead

Since a job could stay in Goose ecosystem for 60+ days, an audit-trail would help with debugging.

The audit-trail can have following details:

• event
• time

• While enqueuing, Clients can specify queue name
• Workers can be initialized with a set of queues

Priority can be tweaked by concurrency of workers.

Because of #14, Goose can either timeout for long times, or shutdown in a clean manner.

If we have a way to interrup connections to redis in a clean manner, Goose will have long-polling & clean graceful-shutdown.

Context

• In past, we've struggled with closing connection pool during graceful shutdown, lack of Redis v6.2.0 support.
• Despite these struggles, after spiking various libraries (link to Redis ADR), we decided to stick with Carmine.

Problem

• Carmine lacks support for Redis Cluster

Solution

• Goose can have a small set of macros for the subset of Redis operations to map keys to the right cluster node. This macro will rewrite Redis calls to follow the cluster protocol of identifying the node and then query it
• If these macros can be generalised, we can raise a PR to Carmine itself

Issue

For multiple worker processes, amplify & randomize polling interval.
This reduces load on redis, approximately achieves configured scheduled queue polling interval despite n workers

Solution

• Blocked on #28
• Sleep for (* poll-interval process-count (rand))

Like perform-in-sec, schedule a job to run recurrently.

For ex, perform-every and take CRON as input.

Implementation details:

1. Add a perfrom-every function to Goose Client that takes a cron expression
2. Calculate next date for the expression, say 3-Aug-1-PM
3. Schedule the job to run at next CRON 3-Aug-1-PM
4. When scheduler finds jobs due for execution, enqueue it to front of queue
5. Alongwith enqueuing, also schedule it back into the queue for recurring execution
6. Enqueue & re-scheduling should happen in 1 transaction so as to not loose the job
7. Define number of times a periodic job should run and then stop
8. Limit :scheduler-polling-interval-sec config to 60s as minimum interval of periodic time is 1 min

Open questions:

1. Should we reuse scheduled-jobs queue for periodic jobs?
2. How we'll calculate latency? Calculate time difference from previous cron schedule?
3. Alongwith standard APIs, add an API to modify CRON period?

• Total process count for statsD & scheduler sleep time
• Add docstrings for cljdocs
• Update API, StatsD & Middleware logic as per Wiki
• Inject error service config into error & death handlers
• Update README & it's badges
• Add prefixed-queue to Job
• Add Redis as default broker

• Enable publisher-confirm mode on client
  • If broker responds with basic.nack for 3 jobs in a row
    • Callback with failed jobs
    • switch to synchronous acks for future jobs
    • switch to async after 5 successful acks
  • Do acks async because latency can be few hundred millis

References

• https://www.rabbitmq.com/tutorials/tutorial-seven-java.html
• https://www.rabbitmq.com/confirms.html#publisher-confirms-latency
• https://www.rabbitmq.com/publishers.html#publisher-confirm-strategies

• Rename Goose Description?
  • s/sidekick for Clojure/Durable background job processing library
• Change integration-test promises to atoms
  • Reset atoms as part of fixtures
• s/prefixed-queue/ready-queue
• Put log-on-exceptions macro inside while-loop macro
• Add more integration tests:
  • heartbeat+orphan-checker
  • Middleware
  • Parallelism: a 5-threaded worker should complete 10 50ms tasks in 110ms
• Improve exception handling in redis namespace
• Re-format function args: either in 1 line or vertically aligned as per clojure style guide
• Modify defs in tests to be within fixtures
• rename fn inside broker protocol: start -> start-worker
• s/:schedule/:schedule-run-at
• s/dead-at/died-at
• Rename metrics.protocol/Protocol to metrics/Metrics
• periodic jobs
  • Return name & schedule when registering a cron-job
  • Accept cron-opts as single config-map in perform-every
  • add config for timezone
• Add util for calculating sleep-duration & sleeping
• Test error-service-config is injected in error & death handlers
• [redis] Test job recovery by orphan-checker

Issue

• How to kill a worker thread from inside a thread?
• Calling .shutdownNow() causes in-progress job to fail and be scheduled for retry
• Kill a worker thread & validate orphan job is re-picked by another worker thread
• This implicitly tests heartbeat

Acceptance Criteria:

• Accepts redis connection params
• Accepts parallelism config
• Accepts backend URL

To be decided...

• Checkout timbre & tools.logging
• Idiomatic/Functional way would be the server injecting a logger & Goose sending events to the logger.
• The interface can be: log('time', 'level', 'msg', & params...)
  • try it out once with tools.logging

Use library: seancorfield/readme
Reference: Nilenso honeysql-postgres

• This will be a first step in direction of supporting RabbitMQ
  The protocol will have following functions:

- enqueue
- schedule
- start
  - reify (goose.worker/stop)
- middleware-chain?
- APIs
    - enqueued
    - dead
    - scheduled

Current interface

(async 
  `fn-sym
   {:args '(1 2 3) 
   :other-opts :other-vals})

Better interface

(async
  opts
 `fn-sym
 "variadic" :args 1.0 2 {:map :val} '("list") ["vector"])

• Client can configure number of worker processes to run in parallel
• Stick to 1 process spawning n threads for now

• When integration tests are run, listen on configured statsd port to verify stats are emitted as expected. Refer this github gist for Datagram listener in Clojure

When registering a periodic job, set a run-count or run-until.
The job will be deregistered from cron-schedule once the run-count has been reached.

nil run-count/run-until means the job will run indefinitely.

Done

• Add ADR to repository
  • Interface
  • Redis client library choice
  • Validations
  • logging
  • using deps, lower version of libraries, etc.
  • reliability
  • priority of queues
  • scheduling
• Add Goose to clojure-toolbox https://github.com/weavejester/clojure-toolbox.com
• Publish to Clojars

Deferred to 0.2

• All exposed functions have a doc string, and present in Clojure docs. Host on cljdocs once done
• Add logo to README
• Flow-chart for scheduler, enqueue, failed jobs, failed jobs with custom queue, dead jobs
• Maintainability Badges
• wiki for every feature
• Changelog

Goose should emit stats for:

• enqueue-execute time diff
• schedule SLA diff
• successful/failed/orphaned/dead jobs

Acceptance Criteria:

• Pull jobs from redis
• Deserialize the arguments and execute the functions
• Gracefully shutdown and enq in-progress jobs back

Stretch goal:

• Multi-threaded workers

Issue

Goose polls redis n times for n threads.
To reduce load on redis, we might want to consider polling from just 1 thread, and enqueuing jobs' execution to the threadpool. To limit execution parallelism/concurrency to the user config, we can have an in-memory buffered queue.

N worker instances with T threads polling means O(N*T) operations per second (or long-polling if #65 gets resolved) slamming Redis.

What next?

• Benchmark 2 approaches:
  • Polling redis n times
  • Polling redis once

While benchmarking, measure 2 things:

• time taken to complete 1000 jobs averaging 50ms execution time
• redis memory/CPU consumption

• By default, Goose sends metrics to statsd
• Have a metrics protocol for users to inject their implementation of metrics

Issue

A count of worker process is needed to coordinate reliable processing of hanging jobs, scheduler-polling time, etc.

Solution

• Generate process-id (based on VM host/container-id + random string)
  • create key with TTL of 1 min
  • Renew TTL every 30 sec
  • Run GC every 1 min to handle abrupt shutdowns
• On startup, add to processes list
• On shutdown, remove from list

• Instead of a random UUID, a job-id can have date+time of enqueuing, and other generic info limited to <60 characters

Outcomes

• Finalize between 2 interfaces: pre-defined jobs using code OR resolving jobs at runtime
• Stats for users
• Comparison for newer releases

Ability to enqueue a job to be picked up at a certain time in future.

Create either a Github wiki, or leverage clj-docs.

It should have following details:

• docstring for public-facing functions
• Broker config, Enqueuing/Dequeuing, Scheduling, Error-handling, API, etc.
• Flow-charts for life-cycles of a job
  • Enqueue-Dequeue
  • Schedule-Enqueue-Dequeue
  • Enqueue-Fail-Schedule-Dequeue
  • Enqueue-Fail-Schedule-Dequeue-from-retry-queue
  • Enqueu-Fail-Dead

• Not all APIs from Redis can be replicated

Acceptance Criteria:

Arguments:

• Takes the job function
• Arguments for the job function

Validations:

• Ensure that the job-fn is serializable
• Ensure that the namespace in the job-fn is present
• Ensure that the args are edn-serializable

Implementation:

• Puts them in redis to be later picked up by the worker

At, present, Goose has 2 options for interfaces:

• Provide fully-qualified & resolvable function symbol
• Predefine functions that can be enqueued, & their retry/schedule configs

Choose between the 2 based on #33

Sometimes, a job execution might lead to worker process being crashed.

Due to orphan-checks, such jobs will be re-enqueued & retried.

Keep a note of recovered jobs, if a job is recovered more than n times, it can be assumed it's poisonous & leads to worker crash.

• Users should be able to inject code pre/post job execution
• Pre/post Job enqueue isn't necessary because of Goose's interface

Issues

Args validation just checks if it is edn serializable. edn serializes anonymous & symbolized functions too, which isn't supported by Goose as they cannot be stored somewhere and retrieved in a different JVM.

Possible solutions:

Try them out, they aren't tested at time of writing

• Check if parents of said object are serializable?
• Define an exhaustive list of allowed types and validate against them
  • Refer https://github.com/ptaoussanis/nippy for the exhaustive list
  • Refer Sidekiq best practices on job params

An API that helps view:

• List failed jobs in retry queue
• List dead jobs (exhausted retries)
• Retry dead jobs
• Retry failed jobs now (instead of later)

Implementation details

• RETRY-EXCHANGE (Re-Enqeueue with delay)
• Use Dead-Letter Exchanges
• https://dzone.com/articles/rabbitmq-consumer-retry-mechanism

• When a job throws an exception, worker re-enqueues with updated retry-count
• Job is retried with an exponential back-off function. User can configure their own back-off function
• users can add an error service like Honeybadger, sentry to report errors on email
• If 0 retries, put in dead-letter queue

Implementation details:

Client

• connection factory
• create a channel. (is should be re-usable across threads)
• create a queue (this operation should be memoized)
  • durable: true, auto-delete: false, exclusive: false
• publish
  • persistent: true, priority: 0

Worker

• connection factory
• create a channel. (is should be re-usable across threads)
  • basic.qos = number of threads
  • global:false (only share across consumer on the channel, not connection)
  • https://www.rabbitmq.com/confirms.html#channel-qos-prefetch
  • https://www.rabbitmq.com/consumer-prefetch.html
• create a queue (only happens on initialization)
  • durable: true, auto-delete: false, exclusive: false
• subscribe
  • manual_ack: true
• channel.ACK
• Set consumer_timeout in rabbitmq.conf
  • https://www.rabbitmq.com/consumers.html#acknowledgement-timeout

Issue

The approach to validation doesn't feel functional. We avoided spec, :pre form in defn, expound, Metis, Validateur because of 2 reasons:

• They simply print error statements. Client doesn't get an explicit error message describing what went wrong
• Customized validations weren't possible

Unsolved requirement

• A perfect solution would mean users generating/understanding function params just by looking at validations

FWIW, Claypoole, a popular Clojure library also does Validations exactly like Goose

Possible solutions

Whichever solution gets chosen, be sure it follows above 2 requirements.

• Wrap spec inside exceptions like this gist

Todo

• Use mocks to assert client/worker validates redis, queue, etc. To avoid duplication, we aren't validating redis from client tests, as redis tests already check that.
• Use table-driven tests for validations

Should be picked up after #105 is completed

Allow clients to register for a callback when a Batch of Jobs is executed
Callback can contain either execution result (success scenario) or exception (failure scenario)

Implementation details

• If schedule-time is in the past, publish with priority: 1
  • Check if this is necessary. i.e. does a negative delay result in job being enqueued to head of queue
• publish(x-delay: 123 ms)

• RMQ sets redeliver to true. It can be used in middlewares, or to send stats
  • https://www.rabbitmq.com/confirms.html#automatic-requeueing
• Heartbeat vs TCP keepalive
  • Detect if TCP keepalive is configured, then Heartbeat can be disabled
  • Set hearbeat timeout to 15 seconds
  • https://www.rabbitmq.com/heartbeats.html#false-positives
• Best Practices Wiki
  • Higher thread-count doesn't necessary imply better throughput. Based on CPU cores & RAM, by doing trial & error - Avoid law of diminishing returns https://blog.rabbitmq.com/posts/2014/04/finding-bottlenecks-with-rabbitmq-3-3/#consumer-utilisation
  • https://www.rabbitmq.com/production-checklist.html
  • https://www.rabbitmq.com/monitoring.html
  • use a cluster

Checklist

• transactions?
• use pool/get-pool & pool/borrow
• should ACK be a middleware? (it's used in 2 places: consumer & retry)
• can we do DLX instead of bury-job?
• use :handle-shutdown-signal-fn inside lc/subscribe
• inject ExecutorService for subscribers
• check if automatically-recover is always true, if not add it
  • recovering from failed conn: http://clojurerabbitmq.info/articles/error_handling.html
• Memoize queue creation & give control to users
• add return-listener
• add ShutdownListener
• add dead-jobs/replay-n-jobs API
• add payload/encode, payload/decode feature inside util
• retry negative acks for sync publisher confirms strategy