-
Openshift 3.9 cluster
-
Istio 0.7.1 with authentication installed on the aforementioned cluster. To install Istio simply follow one of the following docs:
-
Enable automatic sidecar injection for Istio (See this for details)
In order for Istio automatic sidecar injection to work properly, the following Istio configuration needs to be in place:
-
The
policy
field is set todisabled
in theistio-inject
configmap of theistio-system
namespace. This can be checked by inspecting the output ofoc get configmap istio-inject -o jsonpath='{.data.config}' -n istio-system | grep policy
-
The
istio-sidecar-injector
MutatingWebhookConfiguration
should not limit the injection to properly labeled namespaces. If Istio was installed using the default settings, then make sure the output ofoc get MutatingWebhookConfiguration istio-sidecar-injector -o jsonpath='{.webhooks[0].namespaceSelector}' -n istio-system`
is empty. It is advised however that you inspect the output of
oc get MutatingWebhookConfiguration istio-sidecar-injector -o yaml
to make sure that no other "filters" have been applied.
-
-
Login to the cluster with the admin user
Create a new project/namespace on the cluster. This is where your application will be deployed.
oc new-project <whatever valid project name you want>
Execute the following command to build the project and deploy it to OpenShift:
mvn clean package fabric8:deploy -Popenshift
Configuration for FMP may be found both in pom.xml and src/main/fabric8
files/folders.
Run the following commands to apply and execute the OpenShift templates that will configure and deploy the applications:
find . | grep openshiftio | grep application | xargs -n 1 oc apply -f
oc new-app --template=spring-boot-istio-circuit-breaker-greeting -p SOURCE_REPOSITORY_URL=https://github.com/snowdrop/spring-boot-istio-circuit-breaker-booster -p SOURCE_REPOSITORY_REF=master -p SOURCE_REPOSITORY_DIR=greeting-service
oc new-app --template=spring-boot-istio-circuit-breaker-name -p SOURCE_REPOSITORY_URL=https://github.com/snowdrop/spring-boot-istio-circuit-breaker-booster -p SOURCE_REPOSITORY_REF=master -p SOURCE_REPOSITORY_DIR=name-service
Run the following command to determine the appropriate URL to access our demo. Make sure you access the url with the HTTP scheme. HTTPS is NOT enabled by default:
oc get route spring-boot-istio-circuit-breaker-greeting -o jsonpath='http://{.spec.host}{"\n"}'
Open this URL in a web browser.
Try the application to see the default behavior without an Istio configuration.
-
Click on the "Start" button to issue 10 concurrent requests to the name service.
-
Click on the "Stop" button to stop the requests
-
You can change the number of concurrent requests between 1 and 20.
-
All calls go through as expected.
-
Apply the
RouteRule
which is required for theDestinationPolicy
to take effect:oc create -f istio/route_rule.yml -n $(oc project -q)
-
Try the application again.
At this point, nothing should have changed in the application behavior.
-
Now apply the initial
DestinationPolicy
that activates Istio’s Circuit Breaker on the name service, configuring it to allow a maximum of 100 concurrent connections.oc create -f istio/initial_destination_policy.yml -n $(oc project -q)
-
Try the application again.
Since we only make up to 20 concurrent connections, the circuit breaker should not trip.
-
Now apply a more restrictive
DestinationPolicy
, after having remove the initial one:oc delete -f istio/initial_destination_policy.yml oc create -f istio/restrictive_destination_policy.yml -n $(oc project -q)
-
Try the application again
Since the Circuit Breaker is now configured to only allow one concurrent connection and by default we are sending 10 to the name service, we should now see the Circuit Breaker tripping open. However, experimentally, we observe that this does not happen in a clear-cut fashion: the circuit is not always open. In fact, depending on how fast the server on which the application is running, the circuit might not break open at all. The reason for this is that is the name service doesn’t do much and thus responds quite fast. This, in turn, leads to not having much concurrency at all.
We could try to increase contention on the name service using Istio’s fault injection behavior by applying a 1-second delay to 50% of the calls to the name service.
-
Delete the original
RouteRule
and apply a new one to do that:oc delete -f istio/route_rule.yml oc create -f istio/route_rule_with_delay.yml -n $(oc project -q)
-
Try the application again
You should observe that this doesn’t seem to change how often the circuit breaks open. This is due to the fact that the injected delay actually occurs between the services. So, in essence, this only time shifts the requests, only increasing concurrency marginally (due to the fact that only 50% of the requests are delayed). This still doesn’t let us observe the circuit breaking open properly.
-
For more comfort, re-activate the original
RouteRule
since we don’t need the artificial one-second delay:
oc delete -f istio/route_rule_with_delay.yml
oc create -f istio/route_rule.yml -n $(oc project -q)
-
We need to increase contention on the name service in order to have enough concurrent connections to trip open the circuit breaker. We can accomplish this by simulating load on the name service by asking it to introduce a random processing time. To accomplish this:
-
Stop the requests (if that wasn’t already the case)
-
Checking the "Simulate load" checkbox
-
Start the requests.
You should now observe the circuit breaking open.
-