Code Monkey home page Code Monkey logo

connector-postgresql's Introduction

SpiceDB Postgres Connector - Under Construction

GoDoc Discord Server Twitter

connector-postgresql is a tool that translates data from postgres databases into SpiceDB relationships.

No guarantees are made for the stability of the CLI interface or the format of config files.

See CONTRIBUTING.md for instructions on how to contribute and perform common tasks like building the project and running tests.

Getting Started

connector-postgresql import will connect to postgres, generate an example schema and config to map postgres data into SpiceDB, and then attempt to sync that data into SpiceDB as relationships.

By default, it will dry-run to show you what would be synced.

It can also be run as two stages: one to generate an example schema and config, and one to actually import relationships based on that config.

Auto-Import

$ connector-postgresql import --dry-run=false --spicedb-endpoint=localhost:50051 --spicedb-token=somesecretkeyhere --spicedb-insecure=true "postgres://postgres:secret@localhost:5432/mydb?sslmode=disable"
  • Prints out a zed schema + a config mapping from pg to SpiceDB
  • Appends the generated zed schema to SpiceDB's schema
  • Mirrors all relationships into SpiceDB according to that config

Dry-Run

$ connector-postgresql import --spicedb-endpoint=localhost:50051 --spicedb-token=somesecretkeyhere --spicedb-insecure=true "postgres://postgres:secret@localhost:5432/mydb?sslmode=disable"
  • Prints out a zed schema + a config mapping from pg to SpiceDB
  • Logs relationships that would have been written to SpiceDB

Custom Config

$ connector-postgresql import --config=config.yaml --spicedb-endpoint=localhost:50051 --spicedb-token=somesecretkeyhere --spicedb-insecure=true "postgres://postgres:secret@localhost:5432/mydb?sslmode=disable"
  • Uses the provided config.yaml to write relationships into SpiceDB
  • If the required schema is already in SpiceDB, skip appending it with --append-schema=false

Example config.yaml

schema: |2
  definition customer {}

  definition contact {
      relation customer: customer
  }

  definition article {
      relation tags: tags
  }

  definition tags {
      relation article: article
  }
tables:
# for each row in the contacts table
- name: contacts
  relationships:
  # generate a relationship contact:<contact_id>#customer@customer:<customer_id>_<customer_name>
  - resource_type: contact
    resource_id_cols:
    - contact_id
    relation: customer
    subject_type: customer
    subject_id_cols:
    - customer_id
    - customer_name
# for each row in the article_tag table (a join table from articles <-> tags)
- name: article_tag
  relationships:
  # generate a relationship article:<article_id>#tags@tag:<tag_id>
  - resource_type: article
    resource_id_cols:
    - article_id
    relation: tags
    subject_type: tags
    subject_id_cols:
    - tag_id
  # generate a second relationship tags:<tag_id>#article@article:<article_id>
  - resource_type: tags
    resource_id_cols:
    - tag_id
    relation: article
    subject_type: article
    subject_id_cols:
    - article_id

Connect Quickstart

WARNING: This is exploratory, and the current implementation has serious flaws that mean the connector should not be run in production.

The connector can be run continuously with connector-postgresql run. Running the connector will first run a full import via connector-postgresql import, but then it will follow the postgres replication log and sync data as it changes.

$ connector-postgresql run --dry-run=false --spicedb-endpoint=localhost:50051 --spicedb-token=somesecretkeyhere --spicedb-insecure=true "postgres://postgres:secret@localhost:5432/mydb?sslmode=disable"

connector-postgresql's People

Contributors

ecordell avatar jzelinskie avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

connector-postgresql's Issues

Deletes are not properly handled from the generated replication log.

The replication log follower was written assuming that delete events included all column values. But delete events only include the fields referenced by the WHERE clause of delete queries, which means that deletes are not properly propagated to spicedb.

Note that this only affects the run command, which tails the replication log, and not config or import.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.