Code Monkey home page Code Monkey logo

patterns-distributed-systems's Introduction

Resilience Patterns for Distributed systems

Overview

This repository is a compilation of working code scripts that complete the O'Reilly session on Patterns for Distributed Systems The code here will be used during the session to reason and discuss various patterns and their benefits and applications.

The code here can remain an independent and open means of learning about the patterns. This guide outlines how different scenarios be executed and impact of pattern is visualized. The guide covers 5 specific patterns and their execution in the repository.

Tools / Frameworks used

  • Java 11
  • Maven
  • Springboot
  • JMeter as load generation tool
  • Influx db as timeseries store for load metrics
  • Grafana as a visualization tool
  • RabbitMQ as the distributed message queue
  • Docker

Only basic Java knowledge is required to execute the patterns. All other components can be treated as a blackbox.

Application installation

OSX

  • Install Java with brew brew install openjdk@11
  • Install Docker Desktop from here
  • Run docker desktop
  • Install maven with brew brew install maven
  • Clone this repo git clone https://github.com/priyaaank/patterns-distributed-systems.git
  • Setup branches
git branch --track bulkheads origin/bulkheads
git branch --track bulkheads-fix origin/bulkheads-fix
git branch --track transientfailure origin/transientfailure
git branch --track transientfailure-fix origin/transientfailure-fix
git branch --track circuitbreaker origin/circuitbreaker
git branch --track circuitbreaker-fix origin/circuitbreaker-fix
git branch --track gracefuldegradation origin/gracefuldegradation
git branch --track batchtostream origin/batchtostream
git branch --track batchtostream-fix origin/batchtostream-fix
git branch --track backpressure origin/backpressure
git branch --track backpressure-fix origin/backpressure-fix
  • Execute ./rebuild_and_restart.sh
  • Execute following curl command in terminal to validate text, title, etc for url are bring returned
curl -X GET "http://localhost:8080/bookmark/enrich?url=https://github.com&fieldsRequested=title,text,longUrl,shortenedUrl,tags"

Linux (Ubutnu 18.04)

  • cd ~
  • git clone https://github.com/priyaaank/patterns-distributed-systems.git
  • apt-get install openjdk-11-jdk
  • apt-get install maven
  • Follow instructions to install docker
  • Follow instructions to install docker-compose
  • cd ~/patterns-distributed-systems
  • Setup branches
git branch --track bulkheads origin/bulkheads
git branch --track bulkheads-fix origin/bulkheads-fix
git branch --track transientfailure origin/transientfailure
git branch --track transientfailure-fix origin/transientfailure-fix
git branch --track circuitbreaker origin/circuitbreaker
git branch --track circuitbreaker-fix origin/circuitbreaker-fix
git branch --track gracefuldegradation origin/gracefuldegradation
git branch --track batchtostream origin/batchtostream
git branch --track batchtostream-fix origin/batchtostream-fix
git branch --track backpressure origin/backpressure
git branch --track backpressure-fix origin/backpressure-fix
  • Execute ./rebuild_and_restart.sh
  • Execute following curl command in terminal to validate text, title, etc for url are bring returned
curl -X GET "http://localhost:8080/bookmark/enrich?url=https://github.com&fieldsRequested=title,text,longUrl,shortenedUrl,tags"

Windows

  • Help needed to populate the instructions for installation

Load test setup

  • Execute mkdir -p load-testing/mount/
  • cd into the directory cd load-testing/mount/
  • Download JMeter from here
  • Extract files in the folder load-testing/mount/
  • Execute on command line ./apache-jmeter-5.4.3/bin/jmeter
  • Download plugins manager for jmeter from here
  • Move the downloaded jar to load-testing/mount/apache-jmeter-5.4.3/lib/ext/
  • Download influxdb2 metrics reporter jar from here
  • Move the downloaded jar to load-testing/mount/apache-jmeter-5.4.3/lib/ext/
  • Restart Jmeter
  • Open script load-testing/scripts/Bulkheads.jmx
  • If it prompts to install new plugins, click ok and install

Execution

Following is a short guide on executing load tests for specific scenarions and making observations

Bulkheads

Without Bulkheads (Common HTTP thread pool contention)

  • git checkout bulkheads
  • ./rebuild_and_restart.sh
  • Open script ~/patterns-distributed-systems/load-testing/scripts/Bulkheads.jmx and execute in JMeter
  • View Transactions per Second and Summary view

With Bulkheads fix (Individual HTTP thread pools)

Code Fix

  • git checkout bulkheads-fix
  • ./rebuild_and_restart.sh
  • Open script ~/patterns-distributed-systems/load-testing/scripts/Bulkheads.jmx and execute in JMeter
  • View Transactions per Second and Summary view

Circuit Breakers

Without Timeouts (Using RestTemplate without timeout config)

  • git checkout transientfailure
  • ./rebuild_and_restart.sh
  • Open script ~/patterns-distributed-systems/load-testing/scripts/TransientFailure.jmx and execute in JMeter
  • View Transactions per Second and Summary view

With Timeouts fix (Using RestTemplate with timeout config)

  • git checkout transientfailure
  • ./rebuild_and_restart.sh
  • Open script ~/patterns-distributed-systems/load-testing/scripts/TransientFailure.jmx and execute in JMeter
  • View Transactions per Second and Summary view

With Retries (Retry all transient errors)

Code Fix

  • git checkout transientfailure-fix
  • ./rebuild_and_restart.sh
  • Open script ~/patterns-distributed-systems/load-testing/scripts/TransientFailure.jmx and execute in JMeter
  • View Transactions per Second and Summary view

Without Circuit breaker with Systematic failures (Retry causing load on service)

  • git checkout circuitbreaker
  • ./rebuild_and_restart.sh
  • Open script ~/patterns-distributed-systems/load-testing/scripts/CircuitBreaker.jmx and execute in JMeter
  • View Transactions per Second and Summary view

With Circuit breaker for Systematic failures (Circuit breaker preventing load aggravation)

Code Fix

  • git checkout circuitbreaker-fix
  • ./rebuild_and_restart.sh
  • Open script ~/patterns-distributed-systems/load-testing/scripts/CircuitBreaker.jmx and execute in JMeter
  • View Transactions per Second and Summary view

Graceful degradation / Failover (Passing control to failover function for dummy tags)

  • git checkout gracefuldegradation
  • ./rebuild_and_restart.sh
  • Open script ~/patterns-distributed-systems/load-testing/scripts/CircuitBreaker.jmx and execute in JMeter
  • View Transactions per Second and Summary view

Rate Limiting

Without Rate Limiting fix (All messages enqueued by the job locally)

  • git checkout batchtostream
  • ./rebuild_and_restart.sh
  • Open script ~/patterns-distributed-systems/load-testing/scripts/BatchToOLTP.jmx and execute in JMeter
  • View Transactions per Second and Summary view

With Rate Limiting fix (Job queues one record at a time using HTTP API)

Code Fix

  • git checkout batchtostream-fix
  • ./rebuild_and_restart.sh
  • Open script ~/patterns-distributed-systems/load-testing/scripts/BatchToOLTP.jmx and execute in JMeter
  • View Transactions per Second and Summary view

Backpressure

Without Backpressure (Unbounded queue into RabbitMQ Queue)

  • git checkout backpressure
  • ./rebuild_and_restart.sh
  • Open script ~/patterns-distributed-systems/load-testing/scripts/Backpressure.jmx and execute in JMeter
  • View Transactions per Second and Summary view

With Rate Limiting fix (Job queues one record at a time using HTTP API)

Code Fix

  • git checkout backpressure-fix
  • ./rebuild_and_restart.sh
  • Open script ~/patterns-distributed-systems/load-testing/scripts/Backpressure.jmx and execute in JMeter
  • View Transactions per Second and Summary view

patterns-distributed-systems's People

Contributors

priyaaank avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.