Stream a batch of / all responses from the server about typedb-protocol HOT 7 CLOSED

Can you help explain what it looks like from end-user code? How does the API gets called and gRCP methods/messages get called/sent under-the-hood?

But also, the batching behaviour needs to be turned on by default when you do tx.execute(q). That's what I'm expecting this issue to solve. The user can configure batch sizes optionally, but that should be the secondary behaviour.

What I also didn't get from the graph above, is whether each Res is a batch of Iter.Res? Is that so?

The only issue with using large batches for a streaming request is that if the client wants to use the Concept API during the transaction, it must wait until it has received the whole of the last batch.

I think this is okay, given that currently, Grakn's transactions are thread-bound anyways.

from typedb-protocol.

which ensures the next batch is being fetched whilst the previous batch is being consumed

If one tx.execute(q) call gets broken down to multiple batches, I presume that the above should be the case, right?

from typedb-protocol.

In this proposal, does the streaming response have to become async? because as it stands the server can only do a single response per request, i believe. @haikalpribadi i think the proposal isn't to introduce a "message batch" message that is composed of multiple "iter" requests, but rather to stream a bunch of messages from the server in a row...

@adammitchelldev is that right?

from typedb-protocol.

@haikalpribadi

Can you help explain what it looks like from end-user code?

If by end user, you mean user of the grakn client:

The underlying batching is hidden from the user and should happen automatically when the client uses tx.execute, the change is only an addition to the protocol. This way, the behaviour of the protocol changes without requiring all clients to immediately update to this new method (so we can start with client-java and then update other clients more gradually). The client APIs will see no changes (unless we choose to expose batching control settings).

How does the API gets called and gRCP methods/messages get called/sent under-the-hood?

The client sends an Iter.Req message with a batchSize = x or getAll = true. The server then sends back a series of Iter.Res messages. It either sends y messages, where y == x or (y <= x and the last Iter.Res has done == true).

The server implementation iterates the query x times, sending an Iter.Res each time, before responding to further client messages.

When the client sends a request, it remembers x (or getAll = true) and then the RPCIterator pulls a gRPC message each time. When x messages have been pulled, the next iteration triggers a new Iter.Req request and resets x. If an Iter.Res.done == true is pulled, the iterator finishes.

What I also didn't get from the graph above, is whether each Res is a batch of Iter.Res? Is that so?

As describe above, each result of the batch is a separate Iter.Res. This way the results can be received as a stream immediately, making full use of the advantages of gRPC streaming and also requiring no further modifications to the protocol. It's both the easiest way to implement it and also probably the most efficient (since we don't try and re-implement message batching optimisations that gRPC has already implemented).

which ensures the next batch is being fetched whilst the previous batch is being consumed

If one tx.execute(q) call gets broken down to multiple batches, I presume that the above should be the case, right?

Not quite, since in the simple version, the next Iter.Req batch request is triggered only after x messages have been received. This means there is a slight "round-trip" stall after each batch, which adds up if the batches are small and the latency is high. In the case where roundTripLatency == batchProcessingTime, half of your time is wasted. To counter this, you use a very high batch size (like 1,000), so batch processing time is high.

In the double-buffered version, you send 2 Iter.Req.batchSize = x messages, and whenever you receive the x results of the first, you issue a new Iter.Req but continue to pull the Iter.Res from the other request you made. In the case where roundTripLatency == batchProcessingTime, you actually hit 100% efficiency, since the server receives the new Iter.Req right as it finishes responding to the last batch, and the client receives the results from the next Iter.Req right as it finishes receiving results from the last one.

The second advantage here is that now your batch size can theoretically be much lower, leading to lower buffer sizes and faster responsiveness when the client wants to do something different with the stream (as it would currently need to receive all responses before making a request, which could be seen as a separately fixable issue). I think this should be done as an incremental feature on top of the basic version outlined in this issue though.

In the adaptive version, the client adjusts the batch size dynamically to get as close to roundTripLatency == batchProcessingTime as possible, another feature that could be incremental.

from typedb-protocol.

In this proposal, does the streaming response have to become async? because as it stands the server can only do a single response per request, i believe. @haikalpribadi i think the proposal isn't to introduce a "message batch" message that is composed of multiple "iter" requests, but rather to stream a bunch of messages from the server in a row...

@adammitchelldev is that right?

Yes, although I think the current implementation would actually be able to handle this without too much adjustment. The issue would be if additional calls (i.e. to concept API) are made during a tx.stream, since the client would have to flush the last iteration batch before receiving a response to the call. This also introduces the idea that the query is async to the concept API, which may not be desirable.

from typedb-protocol.

Related: https://groups.google.com/forum/#!topic/grpc-io/xbJeiw0BXic
One of the things this proposal relies on is the gRPC layer batching small messages for us. The above issue verifies that as of 2017 the gRPC layer does batch small messages into the same packet, if they get queued fast enough (based on a timer).

from typedb-protocol.

Implemented by #29

from typedb-protocol.