Code Monkey home page Code Monkey logo

pelican's Introduction

pelican

An interface that can implement publishing and consuming messages from various message queue platforms.

Executing the tests

The tests are integration tests and depend on zookeeper and kafka running.

run zookeeper and kafka

$ make run

stop zookeeper and kafka

$ make stop

execute the tests

./gradlew clean test

Configuration

Environment Vars

export MESSAGE_QUEUE_HOST='localhost:9092'

Injection

PelicanAppConfig appConfig = new PelicanAppConfig();
appConfig.setMessageQueueHost("localhost:9092");

Change publish and subscribe settings

In order to change the subscribe and publish settings implement a local PelicanAppConfig and override the methods, propertiesForSubscribe, propertiesForPublish. Then use the local implementation to construct publishers and subscribers.

public class PubSubConfig extends PelicanAppConfig {

    @Override
    public Properties propertiesForSubscribe(String clientId, String consumerGroup) {
        Properties properties = super.propertiesForSubscribe(clientId, consumerGroup);
        properties.put("my.new.prop.key", "my.new.prop.value");
        return properties;
    }

    @Override
    public Properties propertiesForPublish(String clientId) {
        Properties properties = super.propertiesForPublish(clientId);
        properties.put("my.new.prop.key", "my.new.prop.value");
        return properties;
    }
}

How to Publish

PelicanAppConfig appConfig = new PelicanAppConfig();

Map<String, String> message = new HashMap<>();
message.put("test_key", "test_value");

Publish publish = appConfig.publish();

String topic = "test";
publish.send(topic, message);

How to Subscribe

AppConfig appConfig = new AppConfig();


List<String> topics = Arrays.asList("test");
String consumerGroup = "test";

Subscribe subject = appConfig.subscribe(topics, consumerGroup);

Duration timeout = Duration.ofSeconds(100);
List<Map<String, String>> messages = subject.poll(timeout);

// tell the queue the record has been processed.
subject.processed();

Docker

This repo also has the definitions for:

They contain image tags that mirror this project's versions.

The images are to be used for local development only and not production ready.

pelican's People

Contributors

tmackenzie avatar

Watchers

 avatar  avatar

pelican's Issues

Reuse KafkaProducer

In KafkaPublish, make the KafkaProducer an instance variable and allow it to be reused by many instances of KafkaPublish. There are also a few ivars that should be modified to private.

retry

add an interface to publish to a retry/dead-letter topic when unrecoverable errors occur.

config by setting vars

MESSAGE_QUEUE_HOST should be able to be configured by setting a var on PelicanAppConfig

makefile

implement a makefile with docker compose to start kafka, zookeeper...

time out settings, can connect.

allow timeout settings to be configurable.
maybe also add an interface to see if a connection can be made.
Which could be use in health check endpoints.

Log4j2

remove log4j and log4j2 dependencies.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.