Code Monkey home page Code Monkey logo

descartesresearch / coffee Goto Github PK

View Code? Open in Web Editor NEW
0.0 3.0 0.0 474 KB

COFFEE is a benchmarking framework for container orchestrators. It collects metrics on many orchestration tasks, such as container starts, networking, updates, and more. COFFEE supports multiple container orchestration frameworks and aims to have no assumptions on cluster nodes and technology stacks.

License: GNU General Public License v3.0

Dockerfile 0.41% Java 98.57% Shell 1.02%

coffee's Introduction

COFFEE: Benchmarking of Container Orchestration Frameworks

COFFEE (short for Container Orchestration Frameworks' Full Experimental Evaluation) is a benchmarking framework for container orchestration frameworks. It is able to automate repeatable experiments and covers many aspects for modern container orchestration frameworks.

Prerequisites

For development purposes:

  • Java 11 or newer
  • Maven

For actual usage:

  • Docker
  • Test cluster (currently supported: Kubernetes and Nomad)
  • Container image repository accessible in the test cluster

Get started

We provide an overview of COFFEE's components and functionality in our docs. Before build and usage make sure you are familiar with COFFEE's configuration options. After setting all required configuration parameters, consider our guide on how to run an experiment with COFFEE. All necessary build and run scripts can be found in the utils folder of this repository. When something doesn't work, check our troubleshooting doc or don't hesitate to create an issue.

Cite us

Read the full paper on ACM Digital Library

BibTeX entry:

@inproceedings{10.1145/3578244.3583726,
author = {Straesser, Martin and Mathiasch, Jonas and Bauer, Andr\'{e} and Kounev, Samuel},
title = {A Systematic Approach for Benchmarking of Container Orchestration Frameworks},
year = {2023},
isbn = {9798400700682},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3578244.3583726},
doi = {10.1145/3578244.3583726},
booktitle = {Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering},
pages = {187–198},
numpages = {12},
keywords = {container orchestration, nomad, performance, benchmarking, kubernetes},
location = {Coimbra, Portugal},
series = {ICPE '23}
} 

Any questions?

For questions contact Martin Straesser.

coffee's People

Contributors

martinstraesser avatar

Watchers

Jóakim von Kistowski avatar  avatar  avatar

coffee's Issues

Stuck in the demo

Hello,
I tried this program, following the read of the paper.
I got stuck in run.sh ...
My output is this

2023-05-30 00:07:39.437 INFO 2959466 --- [ main] t.d.c.controller.ControllerApplication : Starting ControllerApplication v0.1-SNAPSHOT using Java 11.0.19 on ubuntu with PID 2959466 (/root/COFFEE/controller/target/controller-0.1-SNAPSHOT.jar started by root in /root/COFFEE/utils)
2023-05-30 00:07:39.441 DEBUG 2959466 --- [ main] t.d.c.controller.ControllerApplication : Running with Spring Boot v2.6.4, Spring v5.3.16
2023-05-30 00:07:39.442 INFO 2959466 --- [ main] t.d.c.controller.ControllerApplication : No active profile set, falling back to 1 default profile: "default"
2023-05-30 00:07:40.350 INFO 2959466 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2023-05-30 00:07:40.456 INFO 2959466 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 95 ms. Found 7 JPA repository interfaces.
2023-05-30 00:07:41.243 INFO 2959466 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2023-05-30 00:07:41.259 INFO 2959466 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2023-05-30 00:07:41.259 INFO 2959466 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.58]
2023-05-30 00:07:41.327 INFO 2959466 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2023-05-30 00:07:41.328 INFO 2959466 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1804 ms
2023-05-30 00:07:41.849 INFO 2959466 --- [ main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: default]
2023-05-30 00:07:41.959 INFO 2959466 --- [ main] org.hibernate.Version : HHH000412: Hibernate ORM core version 5.6.5.Final
2023-05-30 00:07:42.174 INFO 2959466 --- [ main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.Final}
2023-05-30 00:07:43.053 INFO 2959466 --- [ main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL57Dialect
2023-05-30 00:07:44.377 INFO 2959466 --- [ main] o.h.e.t.j.p.i.JtaPlatformInitiator : HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
2023-05-30 00:07:44.394 INFO 2959466 --- [ main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2023-05-30 00:07:45.342 INFO 2959466 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2023-05-30 00:07:45.359 INFO 2959466 --- [ main] t.d.c.controller.ControllerApplication : Started ControllerApplication in 6.528 seconds (JVM running for 7.329)
2023-05-30 00:07:48.174 INFO 2959466 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
2023-05-30 00:07:48.175 INFO 2959466 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
2023-05-30 00:07:48.177 INFO 2959466 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 2 ms
2023-05-30 00:07:48.221 INFO 2959466 --- [ Thread-1] t.d.coffee.controller.TestExecutor : Starting parsing script...
2023-05-30 00:07:48.231 INFO 2959466 --- [ Thread-1] t.d.coffee.controller.TestExecutor : Finished script parsing
Start successfulroot@ubuntu:~/COFFEE/utils# 2023-05-30 00:07:49.270 INFO 2959466 --- [ Thread-1] t.d.coffee.controller.TestExecutor : starting the procedures
2023-05-30 00:07:49.273 INFO 2959466 --- [ Thread-1] t.d.coffee.controller.TestExecutor : waiting for all procedures to finish
2023-05-30 00:07:49.281 INFO 2959466 --- [ Thread-3] t.d.c.c.procedure.BaseProcedure : Executing procedure : 1 / 1
2023-05-30 00:07:49.296 INFO 2959466 --- [ Thread-3] t.d.c.c.procedure.ProcedureQueue : adding 1 futures to start queue
2023-05-30 00:07:49.546 INFO 2959466 --- [nio-8080-exec-2] t.d.c.c.m.controller.CommandController : Added 1 command start times to START queue

and never completes.
If i try to manually curl the localhost again I have this:
2023-05-30 00:09:50.046 INFO 2959466 --- [ Thread-4] t.d.coffee.controller.TestExecutor : Starting parsing script...
2023-05-30 00:09:50.051 INFO 2959466 --- [ Thread-4] t.d.coffee.controller.TestExecutor : Finished script parsing
2023-05-30 00:09:50.315 WARN 2959466 --- [ Thread-4] t.d.c.c.o.kubernetes.KubernetesClient : ApiException occured, try reconnect
Exception in thread "Thread-4" java.lang.IllegalStateException: API Exception while accessing createNamespacedDeployment Kubernetes client API:

    at tools.descartes.coffee.controller.orchestrator.kubernetes.KubernetesClient.throwApiException(KubernetesClient.java:284)
    at tools.descartes.coffee.controller.orchestrator.kubernetes.KubernetesClient.callAPI(KubernetesClient.java:275)
    at tools.descartes.coffee.controller.orchestrator.kubernetes.KubernetesClient.createDeployment(KubernetesClient.java:200)
    at tools.descartes.coffee.controller.orchestrator.kubernetes.KubernetesClient.init(KubernetesClient.java:78)
    at tools.descartes.coffee.controller.TestExecutor.startUp(TestExecutor.java:89)
    at tools.descartes.coffee.controller.TestExecutor.run(TestExecutor.java:59)
    at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: io.kubernetes.client.openapi.ApiException:
at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:973)
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:885)
at io.kubernetes.client.openapi.apis.AppsV1Api.createNamespacedDeploymentWithHttpInfo(AppsV1Api.java:683)
at io.kubernetes.client.openapi.apis.AppsV1Api.createNamespacedDeployment(AppsV1Api.java:648)
at tools.descartes.coffee.controller.orchestrator.kubernetes.KubernetesClient.lambda$createDeployment$7(KubernetesClient.java:201)
at tools.descartes.coffee.controller.orchestrator.kubernetes.KubernetesClient.callAPI(KubernetesClient.java:269)
... 5 more

I followed the guide but I am not sure on how to set the following parameters:
ipForLoadAndProxy
all kubernetes.naming
controllerIp

My config is currently this one:

-----------------------------------------

SPECIFIC CONFIGURATION FOR THE CONTROLLER

general settings

controller.exportResults=true
adntroller.reportDirectory=reports
controller.testScript=../test-sequence.script
controller.initialReplicas=0
controller.proxyReplicas=1
controller.networkingTimeoutSeconds=60
controller.database.address=127.0.0.1
controller.database.port=3306

loadgenerator settings

loadgenerator.jarfile=httploadgenerator.jar
loadgenerator.userprofile=loadgenerator/service-user-profile.yaml
loadgenerator.intensityFile=NULL
loadgenerator.requestsPerSec=50
loadgenerator.durationSeconds=6000
loadgenerator.loggingFile=loadout.csv
loadgenerator.requestLoggingFile=loadrequestsout.csv

cluster settings

cluster.orchestrator=kubernetes
cluster.ip=10.96.0.1
cluster.port=443
cluster.controllerIp=10.244.0.100
cluster.proxyNodeName=master
cluster.appContainerPort=8080
cluster.appHealthCheck=true

--- AUTOMATIC CONFIG AT BUILD TIME ---

cluster.appImage=someowner/somerepo:test-app-tag_2022-08-06_15-52
cluster.proxyImage=someowner/somerepo:test-app-tag_proxy_2022-05-24_09-07
cluster.updateImage=someowner/somerepo:test-app-tag_update_2022-08-06_15-54

--- END AUTOMATIC CONFIG AT BUILD TIME ---

kubernetes settings

kubernetes.kubeConfigFile=/root/.kube/config
kubernetes.ipForLoadAndProxy=10.0.2.15
kubernetes.applicationNodePort=30007
kubernetes.proxyNodePort=30006
kubernetes.naming.prefix=of-testing-
kubernetes.naming.proxyDeployment=proxy-deployment
kubernetes.naming.proxyService=proxy-service
kubernetes.naming.proxyLabel=proxy-label
kubernetes.naming.namespace=namespace
kubernetes.naming.label=label
kubernetes.naming.deployment=deployment
kubernetes.naming.service=service
kubernetes.naming.container=container
kubernetes.naming.port=port
kubernetes.update.strategy=RollingUpdate
kubernetes.update.maxSurge=25%
kubernetes.update.maxUnavailable=25%

-----------------------------------------

GENERAL AND SPRING PROPERTIES

logging.level.tools.descartes.coffee.controller=TRACE

server.port=8080

spring.jpa.show-sql=true
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5Dialect
spring.jpa.hibernate.naming.physical-strategy=org.hibernate.cfg.ImprovedNamingStrategy

none: The default for MySQL. No change is made to the database structure.

update: Hibernate changes the database according to the given entity structures.

create: Creates the database every time but does not drop it on close.

create-drop: Creates the database and drops it when SessionFactory closes.

spring.jpa.hibernate.ddl-auto=create

Any clue?
Best.

Post-run result processing

At the moment, results are stored in the database and a summary is printed to the console. For a user it would be nice to have visualization of the results and support for post-processing.

Refactor and extend console reports

Console and CSV exporter/reporter need review and deep refactoring. Not all metrics (e.g. command specifics for update/crash) are not reported.

Support for adaptive load sizing

At the moment, load intensity has to be specified before the experiment. In future, we should support adaptive load, i.e., load should adapt automatically to the test clusters size.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.