smallrye / smallrye-stork Goto Github PK

Built-in gRPC load balancing

In gRPC, load balancing is a bit different than elsewhere.

grpclb

With external load balancer, grpclb, the load balancer provides a list of server addresses and a connection is established with each of the addresses in the list.
Then, e.g. round-robin is used to distribute calls among the connections.

round-robin, pick-first

A list of servers (either from Service Discovery or staticly provided?), connections with all of them are established upfront, then the selected strategy is used to "load-balance" the traffic.

SmallRye Stork vs gRPC

The current model of Stork is to provide either a list of server addresses or a single address, selected by the load balancing strategy.

There are a few ways we could adapt it to gRPC:

• (1) set up connections for all the addresses in the list, use the one selected by the load balancer
  • this seems to be the closest to what is done in gRPC
  • it might need a hook for server addresses refresh. Or the gRPC internals could invoke the Stork ServiceDiscovery periodically
• (2) enhance LoadBalancer to add an option to select a few best server addresses

ATM, I think (1) is the best option, and seems to be the closest to what we have.
Specifically, we should provide a NameResolverProvider backed by Stork ServiceDiscovery and LoadBalancer with Picker based on the Stork LoadBalancer

We need to document the CachingServiceDiscovery and the refresh-period configuration

In Kubernetes service discovery strategy we should retrieve and add the labels from the backend pods for a given service, they could be needed during a custom load balancing strategy. Currently, only the service labels are being added to the service instances.

We had a few asks for failover support in Stork.
Currently, we stand at the position that Stork should only provide service instances and not be involved in making calls, but maybe we should revisit it?

Currently, the way to solve it is to use a failure aware load balancer, such as least-response-time and combine it e.g. with MicroProfile Fault Tolerance @Retry annotation.

Select one service instance and use it until it fails. Then switch to a different one.

Hi There,

Iam using Quarkus 2.6.1 and trying to integrate Stork and failed. The first thing iam not sure ist that I using YAML config files instead of properties.

Is this an problem using Stork with Quarkus?

The second Thing is that I getting the following error during Stork initialization:

No SmallRye Stork ConfigProvider found

Can maybe someone help me?

Best regards

Konrad

Provide a lest-request load-balancing strategy. This strategy tracks the number of inflight requests and selects the instance with the smallest number.

For Eureka, select settings based on this, for others just propagate the attribute to ServiceInstance

services with the same name in mulitple namespaces, check if it w orks

My discovery returns 2 service instances:

• 8080 (running)
• 8090 (not running)

My client application repeatedly calls this service. The first call succeeds, the subsequent calls all fail.

To reproduce:

mvn clean install
java -Dquarkus.http.port=8080 -jar target/quarkus-app/quarkus-run.jar
curl http://localhost:8080/hello/stork => SUCCESS
curl http://localhost:8080/hello/stork => KO
curl http://localhost:8080/hello/stork => KO
curl http://localhost:8080/hello/stork => KO
...

see https://github.com/vsevel/stork-issue/tree/issue_233

Options:

• cache until failure - e.g. for Kubernetes PODs
• refresh periodically
• server-side push

Support global defaults for discovery and load balancer types used across the application:

stork.service-discovery=consul
stork.service-discovery.refresh-period=7M # default value for consul's discovery param refresh period
stork.load-balancer=least-response-time

# hello will use consul and least-response-time by default
hello/mp-rest/url=stork://my-service/hello

# if needed, params can still be used at the service level
stork.my-service.service-discovery.refresh-period=10M # override global default
stork.my-service.service-discovery.application=myapp

# or use an entirely new discovery/load balancer for a specific service
stork.my-service2.service-discovery=kubernetes

Create a custom discovery service that extends CachingServiceDiscovery, integrating with an external service (e.g. consul).
Define a refresh period of 10 seconds or so.
Start the client application and server application and get the client to make a few calls on the server application (it will start by fetching the server url from the external discovery service).
Now shut this external discovery service down, and try to get the client to continue calling the server app.
After the refresh period, an attempt will be made to contact the discovery service, which will fail. As a result all business calls will fail, until the discovery service is restored.
The access to the discovery service should be more resilient (although the discovery service should be very resilient in itself, since it is a central and critical service), especially when services have been successfully fetched at least once.
The proposed enhancement would be to return the previously fetched instances (if any) if fetching new instances fail.
There should be also a configurable read timeout and connect timeout on the discovery. Since getting a valid instance is probably blocking for the client, we do not want to make him wait indefinitely, or even on a large connect or read timeout (e.g. 30 secs), if some instances have been fetched in the past.
Refreshing the instances from the discovery service should be seen as "best effort", when instances have been previously fetched in the past.
On the very first attempt, when there are not cached instances, we may use a bigger read and connect timeouts, because if we fail fetching instances, the business call will fail anyway. So it is worth waiting a little bit longer.
But when instances exist and have been cached, we should wait for a very short period of time only (e.g. 100ms depending on the 90% response time of the external discovery service), and fall back quickly on the cached instances if we do not get an answer in time.

right now Stork reuses serviceName for this purpose, that won't work in some cases.

We now have multiple service discovery providers using a Vert.x instance. At the moment each provider creates its own instance. There are a few problems with this approach:

• the instances are not closed - of course, Stork is intended to be closed when the JVM terminates, but in multi-app environments, it could be a problem;
• each instance has a cost, and lead to the creation of event loop threads

We would need a way for the providers to get access to a potentially managed instance of Vert.x to avoid the two issues.

Right now the type of service discovery for a service is configured as follows:

dux.my-service.service-discovery=consul
dux.my-service.service-discovery.some-service-discovery-property=property-value

As @Ladicek pointed out, it is not good for YAML where users will end up with:

dux:
  my-service:
    service-discovery:
      ~: consul
      some-service-discovery-property: property-value

I think we need to go back to:

dux.my-service.service-discovery.type=consul
dux.my-service.service-discovery.some-service-discovery-property=property-value

The load balancer can be null, and this seems to be handled by the getLoadBalancer() method. However, selectServiceInstance would fail with an NPE in that case.

We may want to push e.g. 99% traffic through v1 and 1% through v2, to implement this in a load balancer, we need info on properties of the service instance.

See http://www.eecs.harvard.edu/~michaelm/NEWWORK/postscripts/twosurvey.pdf.

Provides an option that takes the address-list and shuffles it, so not all the replicas will use the same initial service instance.

Implement configuration abstractions in a core module so that different implementations (e.g. MicroProfile, Quarkus, Spring) could be provided.

Often, services that would use Service Discovery, would also like to register themselves to the Service Registry that is backing up the discovery.

• starting the call, onCallStarted()
• finish the call, onCallCompleted(Throwable th)

Provide the possibility to override the Eureka servlet context path. This corresponds to server.servlet.context-path property.

So far the main user facing API is

ServiceDiscovery {
    Multi<ServiceInstance> getAll(String serviceName)

    Uni<ServiceInstance> get(String serviceName)

    void registerServiceDiscoveryHandler(ServiceDiscoveryHandler handler)

    void registerLoadBalancer(LoadBalancer loadBalancer)
}

We need to decide whether this is satisfactory or if it should be improved.

Caveat: the bom should not bring dependencies from parent

I according to the document "http://smallrye.io/smallrye-stork/1.1.0/custom-service-discovery/" custom "Service Discovery", there is no automatic generating xxxConfiguration class, Additional configuration required?

My parent pom is not “stork-parent”，is that the reason？

When relying on DNS resolution, it can be useful. So, it should be documented.

The initial implementation of a ServiceInstance has a bare minimum of information. In the future it should have much more data such as host, port and schema. It should also have an option to include custom information such as credentials and labels.

it's easier to test CDI beans than SPI providers.
It could be nice to have both, a possibility to add LoadBalancers by creating CDI beans and by SPI

PRovide a very simple selection strategy picking a random instance among the set of instances.

add an option to use service discovery information from multiple providers (e.g. Consul and Eureka) for a single service

The purpose of this enhancement is to be able to define a Service programmatically, as if it had been defined in configuration. This will allow to handle dynamic use cases where the service definitions is only known at run time, or cannot be defined statically in the configuration.

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    public String programmatic(@PathParam("country") String country) {
        Map<String, String> params = new HashMap<>();
        params.put("stork.service.discovery.clock-" + country, "my-disc");
        params.put("stork.service.discovery.clock-" + country + ".country", country);
        Stork.getInstance().defineServiceIfAbsent(params);
        IClock clock = RestClientBuilder.newBuilder().baseUri(URI.create("stork://clock-" + country)).build(IClock.class);
        return "" + clock.currentTime();
    }

Load balancers have important role to route traffic from client to service endpoints. These endpoints might be erroneous in a way that service discovery cannot handle. For example services themselves are not equipped with health checks so service providers cannot ensure their availability.

Utilising fault tolerance's circuit breaker can bring significant value to deal with failing server instances the can be made temporarily as failing.

At the same time load balancer can try different instances available before failing on the service call. This would make the client code less impacted as long as there are still some server instances that can handle the call.

@michalszynkiewicz

Especially, add info about caching

Having it as a class is ok for now, but we may want to use an abstraction here.

Since refresh-period property is not mandatory we should avoid NPE when useing a CachingServiceDiscovery without providing a refresh-period value

similar to reactive messaging connector attributes

Implementing DNS for Kubernetes seems like a good start but can we go further?

This issue is to figure out what are the most common patterns for service discovery and client side load balancing on Kubernetes that Stork can help with.

• add a key-value pairs as kuberneters strategy selectors (label=value)

Currently, the only entrypoint for Stork is a singleton implemented as a static field and accessible via the io.smallrye.stork.Stork.getInstance() static method. This is not very flexible, esp. in the context of integration in a more complex environment. I think that it might be practical to make this more extensible so that an integrator can provide its own way of obtaining the "correct" (and possibly customized) Stork instance.

Furthermore, it might be useful (and not only for testing) to provide a simple builder for to construct a Stork instance, e.g.

Stork stork = Stork.builder()
   .initialize() // service loader
   .addServiceDiscoveryProvider(customProvider) // custom ServiceDiscoveryProvider
   .build();

Can you provide a demo of stork integrating Eurake? Including Eurake Server and Client, thank you very much.

Micrometer integration, probably through an abstraction layer, so that other metrics engines can be integrate too.

• service instances resolved
• service discovery executions
• load balancing selection metrics