apiaryio / ivy Goto Github PK

View Code? Open in Web Editor NEW

9.0 9.0 1.0 179 KB

A Node.js queue library focused on easy, yet flexible task execution.

Home Page: https://www.npmjs.com/package/ivy

License: MIT License

CoffeeScript 99.32% Shell 0.59% Dockerfile 0.08%

sre

ivy's People

Contributors

Stargazers

Watchers

Forkers

nquinlan

ivy's Issues

Kill `Sync` for version 1

Clean up interface: all functions invoked by ivy must take cb as a last argument.

For users, it's easy to wrap and it complicates API internally a lot.

Reopen if you disagree and have areasonable pull request.

Create IronMQ notifier backend

Theoretically, Redis might be unreliable and one can use ironmq pub/sub to notify producer about results.

Use fernet instead of custom aes-256-cbc

Fernet is a standard way to do this,

Provide IronMQ push queue implementation for queue backend

IronMQ feature announcement
node.js library docs

Redis queue backend

Refuse to send a task if it is not in registry

Find a way to introduce continuation by backresolving anonymous functions

[ironmq] After consume all task in queue

I have this errors (ironmq are empty).

IVY_WARNING Cannot retrieve task from IronMQ [Error: {"msg":"internal error"}]

Fix naming

Unify, fix and document naming.

Task results? What does "done" mean for task?

Cannot specify queue type for listen

When listen is called with queue type, it is not changed properly and default Memory queue is used instead.

Add a custom salt to `delayedCall` notification backchannel to guarantee execution on the same node

Ivy crashed after Type Error

Ivy listening for queue messages
ironTask is { id: '6007418770491278636',
  body: '',
  timeout: 60,
  reserved_count: 1,
  push_status: {} }
IVY_IRONMQ_ERROR Retrieve tasks that cannot be parsed as JSON, deleting from queue: [object Object] [SyntaxError: Unexpected end of input]

TypeError: Cannot read property 'name' of undefined
  at /app/node_modules/ivy/lib/queues/ironmq.js:144:27
  at /app/node_modules/ivy/node_modules/iron_mq/lib/client.js:180:18
  at /app/node_modules/ivy/node_modules/iron_mq/lib/client.js:190:18
  at APIClient.Client.parseResponse (/app/node_modules/ivy/node_modules/iron_mq/node_modules/iron_core/lib/client.js:244:16)
  at /app/node_modules/ivy/node_modules/iron_mq/lib/api_client.js:151:16
  at Request._callback (/app/node_modules/ivy/node_modules/iron_mq/node_modules/iron_core/lib/client.js:174:18)
  at Request.self.callback (/app/node_modules/ivy/node_modules/iron_mq/node_modules/iron_core/node_modules/request/index.js:148:22)
  at Request.EventEmitter.emit (events.js:98:17)
  at Request.<anonymous> (/app/node_modules/ivy/node_modules/iron_mq/node_modules/iron_core/node_modules/request/index.js:876:14)
  at Request.EventEmitter.emit (events.js:117:20)
  at IncomingMessage.<anonymous> (/app/node_modules/ivy/node_modules/iron_mq/node_modules/iron_core/node_modules/request/index.js:827:12)
  at IncomingMessage.EventEmitter.emit (events.js:117:20)
  at _stream_readable.js:920:16
  at process._tickDomainCallback (node.js:459:13)

Provide a way to utilize redis connection from application

Redis backend now creates its own connection to Redis for simplicity.

Provide a way to pass one from environment.

Public or Private?

In my opinion private repositories should not be published to npm as public packages. Either don't push to npm or publish the repo.

Ivy consumer crashed

TypeError: Cannot call method 'error' of undefined
  at Ivy.executeTask (/app/node_modules/ivy/lib/ivy.js:119:16)
  at Ivy.scheduledTaskRetrieved (/app/node_modules/ivy/lib/ivy.js:86:17)
  at QueueManager.<anonymous> (/app/node_modules/ivy/lib/ivy.js:28:45)
  at QueueManager.EventEmitter.emit (events.js:95:17)
  at /app/node_modules/ivy/lib/queues/ironmq.js:146:39
  at /app/node_modules/ivy/node_modules/iron_mq/lib/client.js:180:18
  at /app/node_modules/ivy/node_modules/iron_mq/lib/client.js:190:18
  at APIClient.Client.parseResponse (/app/node_modules/ivy/node_modules/iron_mq/node_modules/iron_core/lib/client.js:244:16)
  at /app/node_modules/ivy/node_modules/iron_mq/lib/api_client.js:151:16
  at Request._callback (/app/node_modules/ivy/node_modules/iron_mq/node_modules/iron_core/lib/client.js:174:18)
  at Request.self.callback (/app/node_modules/ivy/node_modules/iron_mq/node_modules/iron_core/node_modules/request/index.js:148:22)
  at Request.EventEmitter.emit (events.js:98:17)
  at Request.<anonymous> (/app/node_modules/ivy/node_modules/iron_mq/node_modules/iron_core/node_modules/request/index.js:876:14)
  at Request.EventEmitter.emit (events.js:117:20)
  at IncomingMessage.<anonymous> (/app/node_modules/ivy/node_modules/iron_mq/node_modules/iron_core/node_modules/request/index.js:827:12)
  at IncomingMessage.EventEmitter.emit (events.js:117:20)
  at _stream_readable.js:919:16
  at process._tickDomainCallback (node.js:463:13)

Provide a way to monitor notification failures in Ivy

This is possible to do on Producer (register task/channel id when send, consider it lost after some time), but it would be nice to encapsulate in Ivy and provide an interface.

IronMQ queue backend

Integration ironmq test fail at master branch

╭─abtris@Ladislavs-MacBook-Pro  ~/Sites/ivy ‹v0.10.26› [git:master]
╰─$ npm test

> [email protected] test /Users/abtris/Sites/ivy
> mocha --reporter spec --timeout 5000 --compilers "coffee:coffee-script/register"



  Test unidirectional task
    When I configure Ivy for both ends
      and register factorial task without callback
        and execute it and wait until it is done
          ✓ task result should not be sent
          ✓ task was executed

  Redis Test Suite
    - Dummy it

  IronMQ Queue Backend Test
    When I configure queue
      and send task into it
        and inspect the queue
          ✓ I should see a task there
          ✓ I should see task scheduled for factorial
      and when I attach consumer
        ✓ queue should be empty (488ms)

  Consuming queue
    When I set up queue with scheduled task
      and I configure consumer and wait for the task to complete
        1) "before all" hook

  Notifiers
    When set up notifier and pause it
      and I send in the task result
        ✓ I can see it in notifier's internal storage
        and when I start consuming notifiers
          ✓ factorial should have been called and counter incremented

  Passing info through queue
    When I call delayed async function on paused queue
      and inspect paused queue
        ✓ I should see a task there
        ✓ I should see task scheduled for factorial
      and when I attach consumer
        and resume queue
          ✓ queue should be empty

  Queue configuration
    IronMQ
      When I listen to IronMQ queue
        ✓ Queue backend should be IronMQ
      When I try to listen to IronMQ queue without giving authentication
        ✓ I should not be able to listen


  12 passing (11s)
  1 pending
  1 failing

  1) Consuming queue When I set up queue with scheduled task and I configure consumer and wait for the task to complete "before all" hook:
     Error: timeout of 5000ms exceeded
    at [object Object].<anonymous> (/Users/abtris/Sites/ivy/node_modules/mocha/lib/runnable.js:175:14)
    at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)




npm ERR! Test failed.  See above for more details.
npm ERR! not ok code 0

  #FIXME: This is tricky to implement as returning task reserves them
  # We can get number of tasks using queue.info
  # For now, use peek at next message
  # As this is test function only, refactor is due