jiajie999 / flasfka Goto Github PK

Push/Pull on Kafka over HTTP

You can install kafka with pip:

pip install kafka

Alternatively, you can build and use the debian package.

To set up flasfka to talk to the Kafka cluster, the following variables are available. Put this in a file (for example /etc/flasfka.conf.py):

# Hosts in the kafka cluster (list of ip:port)
HOSTS=["localhost:9092"]

# You probably don't need to change these

# How long to wait when polling a topic for new messages
CONSUMER_TIMEOUT=0.1

# Maximum of messages returned when consuming from a topic
CONSUMER_LIMIT=100

# When you don't specify a group for listening to a topic, flasfka
# will use this one
DEFAULT_GROUP="flasfka"

The file will be read at flasfka startup, provided it is in the environment variable FLASFKA_CONFIG:

export FLASFKA_CONFIG=/etc/flasfka.conf.py

Running flasfka is then just a matter of launching:

flasfka-serve

By default, it runs on the port 5000. Use flasfka-serve -h to learn how to change that. The debian package wraps flasfka into a uwsgi service, so you don't need to worry about this if you are using it.

Now, assuming flasfka is configured and running:

curl -X POST --data-binary '{"messages": ["msg1", "msg2"]}' "http://127.0.0.1:5000/my-topic/"

This pushes msg1 then msg2 to the topic my-topic.

curl -X POST --data-binary '{"messages": ["msg3"]}' "http://127.0.0.1:5000/my-topic/my-key/"

This pushes msg3 to the topic my-topic with the key my-key.

curl http://127.0.0.1:5000/my-topic/

This retrieves a maximum of 100 of the last messages posted to my-topic, from the default group flasfka.

curl http://127.0.0.1:5000/my-topic/my-group/

This retrieves a maximum of 100 of the last messages posted to my-topic, from the group my-group.

curl http://127.0.0.1:5000/my-topic/?limit=20

This retrieves a maximum of 20 of the last messages posted to my-topic, from the default group flasfka.

Flafska only accepts json data, so your messages need to be in utf-8. If you want to pass arbitrary data, it is recommended to encode them to a suitable format before (e.g. base64).