Delayed LEASE and ignored frames behavior leaves message receipt ambiguous about rsocket HOT 6 CLOSED

Reception of LEASE frame is equivalent to processing of LEASE frame for the protocol processing. As the lease enforcement is done by the protocol for the application, there is no queue delay issue nor any problem from propagation delay from the network. Time is relative to processing of the LEASE.

The LEASE is not a guarantee of processing. It is a "go ahead and try" mechanic. This is a distributed system and thus strict ordering must be externally imposed if desired. With the lease mechanic, it is not assumed nor required. Thus, it is possible for a request to be denied due to changing demands on the Responder side. An ERROR is necessary for load shedding even with a LEASE mechanic and must be handled by the application logic above the protocol.

Granting 1 million requests for 24 hours then changing its mind 2 seconds later is fine. It's probably not advisable, though. A better strategy would be a slow start mechanic or a deli counter ticketing mechanic.
This would minimize the amount of shed load and help ease a thundering herd scenario.

The spec states that request liveness is the application concern. So, the app should handle the possibilities of ERROR as well as RESPONSE (NEXT and COMPLETE, etc.) as it sees fit so all those semantics are up to the app. Does it care or not is the first question.

What specific functionality is desired for the app here? A responder can send ERROR (retry or not is application concern). It can send RESPONSE frames to indicate success (as in send the data). And the Responder has full control of sending LEASEs in a workable strategy to head off having to send too many ERRORs for shedding.

If something is missing, let us (@stevegury , @benjchristensen , and I) know what specifically is missing that would make it easier or better to handle.

from rsocket.

The rule when a responder receives a request frame that exceeds the LEASE it granted to the requester is to ignore the request. That means no ack of any sort is sent back to the requester.

This is like removing ACK from TCP and instead of TCP figuring out if a packet needs retransmission and the receiver figuring out if a packet is a duplicate and automatically doing all that, requiring the sending application using TCP decide what to do.

from rsocket.

Sorry, forgot to add timeout of request at the application level as a possibility. No matter what ReactiveSocket does, a request may never be fulfilled due to application logic (intentionally or not). So, a robust application should timeout a request that doesn't get "answered" within the time that it expects. If ReactiveSocket where to keep a "dead" request alive, that would be just as bad. The protocol implementation has no way to know what the app intended or if the app is even in a good state or not on a per request basis. Just as TCP has no idea either for applications using it.

@stevegury should we make it so that the wording for intended behavior for when a lease is exceeded it SHOULD send back an ERROR for this?

I get your point about lack of reliability for TCP and pushing it up to the application. But in this case, I think it actually is the applications job and it has control of the semantics. ReactiveSocket is just the transport, multiplexing, providing semantics for requests, back pressure, etc.

from rsocket.

should we make it so that the wording for intended behavior for when a lease is exceeded it SHOULD send back an ERROR for this?

I think of this lease as a "soft lease" i.e. the responder reserves the right of rejecting a request after lease expiry but may choose to respond if it has the resources to handle the request. As you mentioned that expecting strict alignment across time boundaries in a distributed systems is not the greatest thing to achieve :), specifically for this usecase. However, in the case where the server is really overloaded as it gave out exact leases, it may be prohibitive to actually send an error response. So, I don't think we should say that RS "SHOULD" send back an error.

from rsocket.

I think that @wmiaw correctly identified a problem here.
It will simplify the implementation if a server has to respond to message received out of a lease scope by an error.

That way, server control the lease window and the number of tickets given to a client, a client will always have an outdated view of that, but that's fine because servers MUST reply with an error if it receives a request "outside" of a lease.

This will also simplify thing like leases rebalancing, or draining (clients will receive an explicit error rather than relying on timeout).

I will update this in #25

from rsocket.

Spec now uses ERROR for request that can not be handled. Also, set of error codes for various use cases and range of error codes.

from rsocket.