liftbridge-io / liftbridge-api Goto Github PK

Would be useful for testing and likely some operational cases.

Copying from Slack:

I thought the behavior of publish on the client implementation docs to be pretty confusing. you pass a subject and it's stated that it's the NATS subject, but it might not be if you decide to use a partitioner/partitionbykey/etc... I feel this would make space for a lot of configuration bugs and may confuse ppl a lot

Proposed solution is to split the publish API in two. One higher level method would receive the stream name and handle partitions through the configuration, while another lower level would just receive the nats subject and publish to it directly.

Was thinking about stream, and subject discovery for administrating liftbridge.

As I understand it NATS has this and it's highly dynamic where you can see the topics and channels being created in real time.

So I want to ask if this is useful to expose over the grpc API.
It will allow building developer and operator tools on top of liftbridge using only grpc. I was thinking about building a simple CUI first.

There are non-gRPC things that we should spec out here:

• NATS message envelope header
• The message encoding which doesn't necessarily need to flow through gRPC
• Maybe in the future, we might spec out a formal on-disk format. Since we're dealing with (I think) log-structured storage, we should be able to get direct replay capabilities to e.g. ingest batches and dump them into a db or something.

Since the scope extends beyond the gRPC interface, should we rename this to liftbridge-api? I don't feel super strongly about this, just a thought.

Make it easy for devs on all OS's to build the GRPC bindings for any language.

I like this as a basis: https://github.com/lucor/fyne-cross
It runs from any developer desktop and runs cross compilation in docker.
It then returns the results ( in this case a built binary) back to the host in the same directory.
The dev has no involvement with docker and the golang code does it all.

People will want to build their grpc implementation in different languages for Servers and desktops and so this provides that.

Its 100% golang, and so its possible to have it call the nodejs, python, c# installers and boot strap correctly for each Language / OS Permutation.

THis will also make it ease to do CI