sczyh30 / vertx-kue Goto Github PK

Vert.x Blueprint Project - Vert.x Kue, a priority task queue powered by Vert.x

License: Apache License 2.0

Java 64.44% JavaScript 26.81% CSS 7.41% HTML 1.35%

vertx redis job-scheduler

vertx-kue's Introduction

Vert.x Kue

Vert.x Kue is a priority task queue developed with Vert.x and backed by Redis. It's a Vert.x implementation version of Automattic/kue.

This blueprint is an introduction to message-based application development using Vert.x.

Detailed Document

Detailed documents and tutorials:

Features

Job priority
Delayed jobs
Process many jobs simultaneously
Job and queue event
Optional retries with backoff
RESTful JSON API
Rich integrated UI (with the help of Automattic/kue's UI)
UI progress indication
Job specific logging
Future-based asynchronous model
Polyglot language support
Powered by Vert.x!

For the detail of the features, please see Vert.x Kue Features.

Build/Run

First build the code:

gradle build -x test

Run in local

Vert.x Kue requires Redis running:

redis-server

Then we can run the example:

java -jar kue-core/build/libs/vertx-blueprint-kue-core.jar -cluster
java -jar kue-http/build/libs/vertx-blueprint-kue-http.jar -cluster
java -jar kue-example/build/libs/vertx-blueprint-kue-example.jar -cluster

Then you can visit http://localhost:8080 to inspect the queue via Kue UI in the browser.

Run with Docker Compose

To run Vert.x Kue with Docker Compose:

docker-compose up --build

Then you can run your applications in the terminal. For example:

java -jar kue-example/build/libs/vertx-blueprint-kue-example.jar -cluster

Architecture

Want to improve this blueprint ?

Forks and PRs are definitely welcome !

vertx-kue's People

Contributors

Stargazers

Watchers

vertx-kue's Issues

Propose the next version - Vert.x Kue 2.0

Planning to propose a next step - Vert.x Kue 2.0. This version is expected to work better in production environment. The architecture of Vert.x Kue should be refactored and refined. Here is some of expected features:

Adapt to large scale data processing
Separate the scheduler (more generalized)
Better mechanism dealing with failure
Support more distributed storage system (e.g. HDFS, Ceph)
Enhance monitoring
Performance Optimization

Incorrect behavior during the job backoff

This was due to incorrect state of job-specific message consumer.

Integrate backoff retry with Vert.x Circuit Breaker

This is a good chance to introduce how and where to use Vert.x Circuit Breaker.

Improve extendibility for task adapter, scheduler and backend

The architecture should be refactored to improve extendibility:

Task definition can be customized
Task scheduler interface and common implementations (e.g. priority-based)
Multiple task storage backend

Build warnings

Anyone else getting these warning during the kue-core build?

kue-core\src\main\java\io\vertx\blueprint\kue\CallbackKue.java:20: warning: Interface io.vertx.blueprint.kue.CallbackKue does not redeclare the @Fluent return type  of method getWorkTime(io.vertx.core.Handler<io.vertx.core.AsyncResult<java.lang.Long>>) declared by io.vertx.blueprint.kue.service.JobService
public interface CallbackKue extends JobService {

...one for each @Fluent-attributed method in the CallbackKue interface.

Sometimes throw java.lang.ClassCastException

Sometimes the application sometimes would throw java.lang.ClassCastException: java.lang.String cannot be cast to [Lio.vertx.redis.impl.Reply like this(version: 3.3.0-SNAPSHOT):

六月 11, 2016 7:47:55 下午 io.vertx.core.impl.ContextImpl
严重: Unhandled exception
java.lang.ClassCastException: java.lang.String cannot be cast to [Lio.vertx.redis.impl.Reply;
    at io.vertx.redis.impl.Reply.asType(Reply.java:124)
    at io.vertx.redis.impl.Reply.asType(Reply.java:156)
    at io.vertx.redis.impl.RedisConnection.handleReply(RedisConnection.java:376)
    at io.vertx.redis.impl.ReplyParser.handle(ReplyParser.java:180)
    at io.vertx.redis.impl.ReplyParser.handle(ReplyParser.java:21)
    at io.vertx.core.net.impl.NetSocketImpl.handleDataReceived(NetSocketImpl.java:312)
    at io.vertx.core.net.impl.VertxNetHandler.lambda$channelRead$24(VertxNetHandler.java:67)
    at io.vertx.core.impl.ContextImpl.lambda$wrapTask$3(ContextImpl.java:357)
    at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:234)
    at io.vertx.core.net.impl.VertxNetHandler.channelRead(VertxNetHandler.java:67)
    at io.vertx.core.net.impl.VertxNetHandler.channelRead(VertxNetHandler.java:31)
    at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:122)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:335)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:327)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1336)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:335)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:544)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:485)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:399)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:371)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)
    at java.lang.Thread.run(Thread.java:745)

So far I haven't found out where the Exception throws. I'll try to repeat the exception and fix it.