cuulee / kafka-streams Goto Github PK

This project forked from nodefluent/kafka-streams

0.0 1.0 0.0 577 KB

equivalent to kafka-streams :octopus: for nodejs :sparkles::turtle::rocket::sparkles:

License: MIT License

JavaScript 99.72% Shell 0.28%

kafka-streams's Introduction

node-kafka-streams

npm install --save kafka-streams

const {KafkaStreams} = require("kafka-streams");

const config = require("./config.json");
const factory = new KafkaStreams(config);

const kstream = factory.getKStream("input-topic");
const ktable = factory.getKTable(..);

kstream.merge(ktable).filter(..).map(..).reduce(..).to("output-topic");

API Overview

README Overview

Prerequisites
Aim of this Library
Description
Port Progress Overview
Operator Implementations
Additional Operators
Stream Action Implementations
Join Operators Status
Window Operations
FAQ - More

Prerequisites

kafka broker should be version >= 0.9.x
nodejs should be version >= 6.10

Aim of this Library

this is not a 1:1 port of the official JAVA kafka-streams
the goal of this project is to give at least the same options to a nodejs developer that kafka-streams provides for JVM developers
stream-state processing, table representation, joins, aggregate etc. I am aiming for the easiest api access possible checkout the word count example

Description

kafka-streams 🐙 equivalent for nodejs ✨🐢🚀✨ build on super fast 🔥 observables using most.js 🤘

ships with sinek 🙏 for backpressure

the lib also comes with a few window operations that are more similar to Apache Flink, yet they still feel natural in this api :squirrel:

overwriteable local-storage solution allows for any kind of datastore e.g. RocksDB, Redis, Postgres..

async (Promises) and sync stream operators e.g. stream$.map() or stream$.asyncMap()

super easy API

the lib is based on sinek, which is based on kafka-node's ConsumerGroups therefore it still requires a zookeeper connection (dont worry, your offset will be stored in the kafka broker)