textileio / go-threads Goto Github PK

This should eventually be configurable

AddFollower should work like,

• Get or create a symmetric share key for the thread (you will have it if you or anyone else has already added a follower). Store the key in the metadata store because it's for the whole thread.
• Use this key to encrypt all log read keys before sending them to the follower.
• Tag the key on to the thread address when sharing it.

Now, on the client,

• Do NOT send the share key to the follower, just use it to decrypt the log read keys

There're some tests for Metadata but in the Threadservice suite scope that uses inmem implementation.

I'll add some tests to a metadata_suite.go to test both inmem and tstoreds. Making it in other PR to avoid blocking the current one since it is making lots of changes.

Make a ipfs go-datastore implementation for AddrBook

Components:

• go-textile-threads
• go-textile-wallet
• go-textile-core (to replace go-textile-crypto w/ proto types)

Think about a bit more about how to manage error handling.

In progress here: textileio/go-textile#855

This assumes that key rotation will happen at the thread level.

Make a ipfs go-datastore implementation for Keybook

Use ipfs go-datastore implementation for ThreadMetadata in Threadstore

Connecting with go-textile-threads will uncover most decisions regarding eventstore.Event vs threads.Event transformation.

Users should not need to care about each log - they should just pull at the thread level. We can move the current Pull method internally as it will still be needed.

We'll need a daemon that runs a server and a gRPC service that can control multiple stores.

RPC Methods

• NewStore
• RegisterSchema
• Start
• StartFromAddress
• ModelCreate
• ModelHas
• ModelSave
• ModelDelete
• ModelReadTxn
• ModelWriteTxn
• ModelFindByID
• ListenForStateChanges?
• others?

The threads protocol + event store / model registry.

Currently, there's no mechanism for saying, "this is an invite to be a follower, which means that your address will be advertised to others".

Add, Put, and Pull can all result in new records being added - they can be pushed over a channel.

Requires #386

Possibly with Docker compose, run a thread tests with a large number of participants and high write load.

This is used in conjunction with thread links to facilitate adding a new thread.

As @jsign mentioned in a review, we should avoid have multiple pull jobs running for a single thread.

Idea: Maybe adding extra logic to avoid pulling concurrently from the same thread.

Like, if t.Pull is slow enough and tick ticks again, maybe pulls for the same thread start queueing. If that makes sense there're could be multiple options: from discarding it since there's an already running one, or cancelling running one and retry from zero again, etc.

This can give more guarantees that at most there are 1 live pull for the existing threads (which also in the future we can think to move it to a (capped) worker model to cap it if that's too much pressure).

At the very least we could have a sync.Map or equivalent that tracks live pulls.

Current setup is HTTP + a libp2p adapter. However, since many payloads contain raw bytes, having a serialization layer that can handle typed bytes (protos) would be ideal.

I'm considering gRPC + libp2 transport (I have an example of this working nicely). Pubsub is another consideration. Also considering gRPC.

Requires #386

When all things are wired, make a simple but useful demo app to share.

As mentioned in textileio/go-eventstore#7, we should decide on something if an ongoing dispatch of a set of events fails.

The problem is quite complex, but not really a blocker to get going. Even if no hard decision is made, maybe proper documentation may be enough.

Some summary/ideas to have a sense of the problem, complexity, and possible solutions.

Problem

When a transaction is committed, all generated events from the actions performed in the transaction are dispatched to the Dispatcher. On every event dispatch, the Dispatcher calls the Reduce function of every registered reducer. If any calls to a registered reducer fail, the Dispatch() operations fail.

Consequence 1

The event that caused the failure may have gone through half of the dispatches before failing. This means that half of the models reacted to the event, and the other half didn't.

Consequence 2

Before the event that caused the failure, other events could have been completely processed correctly by all reducers. Similarly, events that would be processed after the event that caused the failure didn't go through the dispatcher.

Summary

Both consequences are an inconsistent state at two different levels (partial event dispatch, and complete and incomplete dispatch of other events in the txn).
The concept of Rollback sounds appealing, but that implies more work to ask in Models.
Possible ideas leveraging the event sourcing mechanism:

• On failure reapply history of events up to the failed event (costly, but would work)
• Every Reduce processing of an event, should store some kind of rollback action to be prepared if that is needed. (Not sure about is possible in all cases to have an undo-event concept)
• Every Model can have a snapshot of its state before applying the last event, just in case is needed for a rollback (can leverage an existing snapshot feature in datastore? if not, not sure the work to implement that outweighs the benefits)
• Consider @carsonfarmer insight of leveraging Action.Previous to have previous state, and try to play with past Action to rollback things. It sounds a good idea.

Make a ipfs go-datastore implementation for HeadBook

Read and test related concepts and libraries that possibly will be part of the complete Eventstore

Do some tests to use QUIC as a transport layer

As mentioned by @carsonfarmer
Consider it as a replacement for the currently underlying JSON schema and encoding.
Notes:

• pro: smaller encoded values
• pro: faster validation of instance with schema?
• pro: will be interoperable with other languages
• cons: would make certain EventCodec unusable: json-patcher, json-crdt, etc

Quoting @carsonfarmer in textileio/go-eventstore#7 :

Do you still think having time first is a good idea? I was starting to wonder if prefix searching via entity it would be useful. Then you'd want something like ///?

Not clear now for me when we're going to query raw events from dispatcher, so no clear answer.
Leaving here to keep record of the question because might be important to reconsdier.

Massage the existing thread components in go-textile into the new interface implementations.

Daemon based, manage multiple logs per thread

• #54 JWTs used to enable remote log writing
• #55 Serve and push over a web-socket at the thread level

Fix dependencies and code to support Go 1.13

Allow a remote client to read/write via a JWT + role claim.

Allow a thread to be followed without knowledge of its secret. This enables more "public" style threads, e.g., a blog, twitter, etc.

We'll need a web socket server and some way of authenticating, most likely with a JWT.

re/ #54

Currently there's a JsonPatcher implementation for creating events. Ultimately, using rdoc (or similar since since other languages requiriements are important) will be ideal.

For some reason, the tests peers can usually not find each other when run on CircleCI.