fgribreau / postgresql-to-amqp Goto Github PK

View Code? Open in Web Editor NEW

30.0 3.0 1.0 25 KB

PostgreSQL to AMQP, forward PostgreSQL notifications to an AMQP queue.

Home Page: https://twitter.com/FGRibreau

Rust 98.01% Dockerfile 1.99%

postgresql amqp rabbitmq events pgnotify crates amqp-queue notifications

postgresql-to-amqp's Introduction

⚠️ Discontinued

@subzerocloud took inspiration from this project and built pg-amqp-bridge which now has more options, use it instead 👍

🚇 PostgreSQL to AMQP gateway

Forward PostgreSQL `pg_notify` notifications to an AMQP queue.

⛴ Cargo

cargo install postgresql-to-amqp

🐳 Docker

docker run --rm -it \
-e POSTGRESQL_URI=postgresql://username:[email protected]:port/database \
-e POSTGRESQL_CHANNEL=foo \
-e AMQP_URI=amqp://127.0.0.1:5672/ \
-e AMQP_QUEUE_NAME=queueName fgribreau/postgresql-to-amqp

⚙ Configuration

Configuration is done through environment variables:

POSTGRESQL_URI: e.g. postgresql://username:[email protected]:port/database
POSTGRESQL_CHANNEL: e.g. foo
AMQP_URI: e.g. amqp://127.0.0.1:5672/
AMQP_QUEUE_NAME: e.g. queueName

🎩 Usage

Start the forwarder:

POSTGRESQL_URI="postgresql://username:[email protected]:port/database" POSTGRESQL_CHANNEL="foo" AMQP_URI="amqp://127.0.0.1:5672/" AMQP_QUEUE_NAME="queueName" postgresql-to-amqp

Execute in psql:

SELECT pg_notify('foo', 'payload');

The forwarder will log and forward the notification to the amqp queue:

Forwarding Notification { process_id: 31694, channel: "foo", payload: "payload" } to queue "queueName"

👁 Philosophy

Low memory consumption (1,9Mo)
Single binary
No dependency
Predictable performance

🔫 Todo

I will happily accept PRs for this:

Related work

pgsql-listen-exchange - RabbitMQ Exchange that publishes messages received from PostgreSQL Notifications

postgresql-to-amqp's People

Contributors

Stargazers

Watchers

Forkers

keruspe

postgresql-to-amqp's Issues

A few ideas (if you want to consider)

I see this as a "bridge" between the two components, so with this idea in mind, why not have the convention that the PG channel is equal to the RMQ entity, where entity is either an exchange or a queue. This is also probably the way users will want to set up things anyway, in order to make it obvious where the PG event will end up in RabbitMQ.

I am not sure if this component should take on the creation of the queue/exchange since this will complicate the code and the config interface, maybe it should just error out (or try to reconnect again) if the entity is not yet created in Rabbit.

Another idea to consider is that you probably want to be able to bridge multiple channels at the same time (each handled by a separate thread)

With this ideas/features in place, the config interface is very simple yet very flexible/powerful

docker run --rm -it \
-e POSTGRESQL_URI=postgresql://username:[email protected]:port/database \
-e AMQP_HOST_PORT=127.0.0.1:5672 \
-e BRIDGE_CHANNELS="foo:queue,bar:exchange" \
fgribreau/postgresql-to-amqp

What do you think?

AMQP_HOST_PORT format issue

Hi. I'm experiencing an issue with setting AMQP_HOST_PORT to FQDN instead of IP.

$ docker logs rabbitmqexample_postgresql-to-amqp_1
    Finished release [optimized] target(s) in 0.6 secs
     Running `target/release/postgresql-to-amqp`
thread 'main' panicked at 'amqp_host_port should be in format '127.0.0.1:5672': AddrParseError(())', /checkout/src/libcore/result.rs:859
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Here is sample docker-compose.yml file to reproduce the issue. Save it and run docker-compose up and you will see the same error message as above.

version: '2'
services:
  postgresql:
    image: postgres:9.5.3
    ports:
      - "5432"
    depends_on:
      - rabbitmq
    environment:
      - POSTGRES_PASSWORD=password
      - POSTGRES_USER=postgres
      - POSTGRES_DATABASE=test
  rabbitmq:
    image: rabbitmq:3.6.9-alpine
    hostname: test-rabbitmq
    mem_limit: 256m
    ports:
      - "5672"
  postgresql-to-amqp:
    image: fgribreau/postgresql-to-amqp:0.1.3
    environment:
      - POSTGRESQL_URI="postgresql://postgres:password@postgresql:5432/test"
      - POSTGRESQL_CHANNEL="mychannel"
      - AMQP_HOST_PORT="rabbitmq:5672"
      - AMQP_QUEUE_NAME="myqueue"
    depends_on:
      - rabbitmq
      - postgresql