Code Monkey home page Code Monkey logo

meson's Introduction

meson

Build StatusClojars ProjectClojure version

Clojure Client Library for the Mesos HTTP API

meson noun. A functional particle composed of a Mesos deployment and a Mesos HTTP API service bound together by the strongly interacting force of Clojure.

Contents

About

This project is a work in progress.

Meson aims to provide an HTTP-only Clojure client API for Mesos with no dependencies upon the Mesos Java library nor protobufs, thus making installation and dependencies much easier to manage.

Dependencies

  • Java
  • Mesos 1.3+
    • You need a running Mesos deployment
  • lein

Versions

Meson Version Meson Status Mesos Clojure Java
0.4.0 (beta) future ??? ??? ???
0.3.0 (alpha) future ??? ??? ???
0.2.0-SNAPSHOT (prototype) in progress 1.3 1.8.0 8 (build 1.8.0_91-b14)
0.1.0-SNAPSHOT (prototype) released 1.0 1.8.0 8 (build 1.8.0_91-b14)

Quick Start

Meson provides some lein commands you can use while developing in the library:

  • lein meson mesos start
  • lein meson mesos stop
  • lein meson mesos restart

These may also be used from the Meson REPL (using the dev profile):

(require '[meson.ops.mesos :as mesos])
(mesos/start-local-docker)
(mesos/stop-local-docker)
(mesos/restart-local-docker)

These will start a Docker container running with locahost ports 5050 and 5051 mapped to those of the running Mesos container.

Documentation

Meson API Reference docs are available here:

Usage

Provide Framework, Use Default Handlers and Default/Localhost Mesos

Subscribe a framework with default handlers to a localhost Mesos master.

Purpose: Development/testing.

Start up the REPL (and be sure to have a running Mesos deployment), then:

(require '[clojure.core.async :as async]
         '[clojure.pprint :refer [pprint]]
         '[meson.api.scheduler.core :as scheduler])
(def framework-info {:framework-info
                     {:user "user1"
                      :name "a-framework"}})
(def channel (scheduler/subscribe framework-info))

Since subscribe was only called with the framework setup info and didn't also pass a handler function or mesos master URL, it will use the default framework handler (basically just a logging message for each scheduler event) and subscribe to localhost. Any real-world Meson framework will have its own handler, pass it to the subscribe function, and also pass a Mesos master scheduler api URL.

Provide Framework and Handlers, Use Default/Localhost Mesos

Subscribe a framework, providing your own handler functions, to a localhost Mesos master.

Purpose: Development/testing.

  • Define your own handler functions as multi-methods. (See meson.api.scheduler.handlers for an example.)
  • Same requires and framework definition as previous section.
  • Subscribe command is now:
(def channel (scheduler/subscribe framework-info my-handlers))

Provide Framework, Handlers, and Mesos Master

Subscribe a framework, providing your own handler functions, to a remote Mesos master.

Purpose: Production use.

  • Define your own handler functions as multi-methods. (See meson.api.scheduler.handlers for an example.)
  • Same requires and framework definition as previous section.
  • Retrieve the URL to your remote Mesos master scheduler api.
  • Subscribe command is now:
(def channel (scheduler/subscribe framework-info my-handlers "https://my-mesos-master.org:5050/api/v1/scheduler"))

Provide Framework, Mesos Master, Use Default Handlers

Subscribe a framework, with default handler functions, to a remote Mesos master.

This is a scenario in which you want to verify you can successfully subscribe to a remote mesos master, but have not written handlers yet.

CAUTION: The default handlers do NOT decline offers. Once subscribed, all offers will go to the framework and wait, preventing other frameworks from getting offers.

Purpose: Development/testing.

  • Require an additional meson library. (see code below)
  • Same framework definition as previous section.
  • Retrieve the URL to your remote Mesos master scheduler api.
  • Subscribe command is now:
(require '[clojure.core.async :as async]
         '[clojure.pprint :refer [pprint]]
         '[meson.api.scheduler.core :as scheduler]
         '[meson.api.scheduler.handlers :as scheduler-handlers])
(def channel (scheduler/subscribe framework-info scheduler-handlers/default "https://my-mesos-master.org:5050/api/v1/scheduler"))

Subscription Results

Regardless, a channel is returned in both cases, and that can be interacted with directly in the REPL, for example:

(def result (async/<!! channel))
(pprint result)

Which should give something like:

{:subscribed
 {:framework-id {:value "6919832b-083e-4db5-89ae-a0d1d222510a-0016"},
  :heartbeat-interval-seconds 15.0,
  :master-info
  {:address {:hostname "0414d10a4f36", :ip "172.17.0.2", :port 5050},
   :hostname "0414d10a4f36",
   :id "6919832b-083e-4db5-89ae-a0d1d222510a",
   :ip 33558956,
   :pid "[email protected]:5050",
   :port 5050,
   :version "1.3.1"}},
 :type :subscribed}

License

Copyright © 2016-2020, Clojure-Aided Enrichment Center

Apache License, Version 2.0.

meson's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

meson's Issues

Add system tests

System tests for Meson will take the form of a Mesos/Meson framework with appropriate assertions.

Depends upon: #21
Satisfies skipped item in #39

Devise an appropriate means for reading stream data from a long-lived HTTP connection

Decide upon an initial project layout

I'm thinking of something like this:

  • meson/
    • core.clj
    • http.clj
    • util.clj
    • scheduler.clj
    • scheduler/
      • ...
    • executor.clj
    • executor/
      • ...
    • operator.clj
    • operator/
      • ...

Create example unit, integration, and system tests

This will utilize the new Mesos testing infrastructure.

Depends upon:

Tasks:

  • Update existing unit tests with ^unit annotations.
  • Create a working integration test

Skipping:

  • Create a working sample system test

Background

We are using the following definitions (from the Wikipedia article on Software Testing):

  • Unit Tests - verify the functionality of a specific section of code
  • Integration Tests - verify the interfaces between components against a software design
  • System Tests - verify that the complete system -- software, dependencies, services -- meets its operational requirements

Define clients data structure

There will be four client data structures we need to define:

  • Base client data
  • Scheduler client data
  • Executor client data
  • Operator client data

Each of the latter three will differ based upon the individual APIs they represent, with the first one defining all the data common to the rest.

The client data that will be used by the library functions will need to have the following defined for it:

  • base url
  • version
  • function for building base url + version
  • command connection
  • event/subscription connection
  • options (some of which might get passed on to clj-http)
  • client-specific data
    • base client
    • scheduler
    • executor
    • operator

Probably just start simply with a record with a few fields and some protocols with a few methods, and add more later as needed.

Fix failing tests

This is due to a difference in user agent string (based on Java version) between dev environments and Travis CI environment. Let's do a regex in the test to fix this ...

Implement Agent API

Functions required:

  • Agent Protocol
    • get-container-status
    • get-resource-status
  • Config Protocol
    • get-version
    • get-flags
  • Debug Protocol
    • start-profiler
    • stop-profiler
  • Files Protocol
    • browse
    • debug
    • download
    • read
  • Health Protocol
    • get-health
    • get-metrics
    • get-state
    • get-system-stats

Part of epic #2

Add RecordIO parsing

The scheduler sends results in RecordIO format (I think this is from Go? It's used internally at Google, and they haven't released a standalone library yet). We need to be able to parse that.

Files/URLs of potential interest:

The last bullet provides example Python which has since been converted to pseudo code on the HTTP docs site.

Related commit to protobuf:

Implement Executor API protobuf types

The executor API protobuf types are likely already available due to the copy from Mesomatic. However, they may not be organized properly into an appropriate namespace. They also might need updates for the 1.x API from Mesos ...

This ticket depends upon the types namespace reorg ticket: #29

Implement Executor Environment Vars

  • MESOS_FRAMEWORK_ID
  • MESOS_EXECUTOR_ID
  • MESOS_DIRECTORY
  • MESOS_SANDBOX
  • MESOS_AGENT_ENDPOINT
  • MESOS_CHECKPOINT
  • MESOS_EXECUTOR_SHUTDOWN_GRACE_PERIOD
  • MESOS_RECOVERY_TIMEOUT
  • MESOS_SUBSCRIPTION_BACKOFF_MAX

Implement Scheduler Calls

Sub-tasks:

  • SUBSCRIBE
  • TEARDOWN
  • ACCEPT
  • DECLINE
  • REVIVE
  • KILL
  • SHUTDOWN
  • ACKNOWLEDGE
  • RECONCILE
  • MESSAGE
  • REQUEST

Implement Scheduler API JSON types

The scheduler API protobuf types are likely already available due to the copy from Mesomatic. However, they may not be organized properly into an appropriate namespace.

Also, there may need to be a base-level, general-use namespace for protobuf types and/or related code that will be usable by all API protobufs (that is now its own ticket: #29)

Define a best-practice and mechanism for event handlers

This will essentially be how developers will define Mesos HTTP APIs in Meson for events. In particular, it needs to provide an example (e.g., defmulti and defmethods) for how to dispatch callbacks based on a message type parsed from the stream.

Subtask of feature #56.

Support API event streams

The Mesos HTTP API requires support for long-lived connections to the scheduler and executor in order to stream events for each of these. In addition, the new operator API also publishes events via long-lived HTTP connections.

Tasks:

  • Devise an appropriate means for reading stream data from a long-lived HTTP connection - #51
  • Create a namespace and functions for parsing streamed data - #52
  • Define a best-practice and mechanism for event handlers - #53
  • Create integration tests for streaming connections - #54

Wont do:

  • Create integration tests for event handling - #55

The reason that #55 won't be done as part of this feature is that the code for the event handling will actually be landing in different tickets for each individual HTTP API event handler.

Part of epic #2

Setup Travis CI Docker to install and run Mesos

Hopefully this can be done without compiling Mesos from scratch on the Docker image ...

Tasks:

  • identify binary install suitable for debian-based Travis CI docker image
  • update .travis.yml to install this binary
  • update .travis.yml to start Mesos master and agent

Ditch protobufs for serialization; use JSON instead

We're going to support only JSON for this library. However, we will provide the ability of using protobufs (and converting those to JSON) for those who want to using existing types from Mesomatic.

Tasks:

  • Move Mesomatic protobuf type conversion functions into meson.mesomatic - #60
  • Change protobuf namespace to json - #61
  • Convert existing protobuf-centric records to json-centric records - #62
  • Remove unnecessary protobuf code - #63

Part of epic #2

Figure out a mechanism for handling multiple client versions

The Mesos HTTP client uses the API path to set the client version, e.g. /api/v1/scheduler.

We want to be able to call a different version of the API with any given client function, but to use the default if none is specified by the developer.

Hrm, if we use records for clients, the client data would supply the default, and if a :version x.y was passed to a client function, that passed version would be used instead of the version specified by the client data. This would mean that a context (base URL, essentially) needs be built with every call (no big deal).

Utilize TravisCI Mesos dockerfile for local testing

Not sure what's involved here, but it would be good to do the same testing locally ... @bjorstrom I'll leave it to you to add the necessary tasks.

Depends upon #37.

Tasks:

  • Identify the docker image utilized by travis ci
  • provide a script for pulling down the image
  • create local travis scripts for setting up the host environment
  • create local travis scripts for setting up the guest environment
  • deps like is done with the .travis.yml file
  • create a Dockerfile that runs the scripts that are meant to operate on the guest
    • update apt repos in Dockerfile
    • work around upstart/docker issues
    • remove extra postgres installs
    • upgrade from 12.04 to 14.04
    • install mesos
    • start mesos
    • set up command for running tests inside the guest
  • create make targets for easily running local travis testing

Setup testing infrastructure

Feature tasks:

  • Add testing profile - #34
  • Fix failing tests - #27
  • Fix lint errors - #41
  • Setup Travis CI Docker to install and run Mesos - #37
  • Utilize TravisCI Mesos dockerfile for local testing - #38
  • Create example unit, integration, and system tests - #39

Part of epic #2

Update namespaces for Meson Client, Scheduler, and Executor APIs

Given the latest changes to the codebase (the new protocols and behaviours and the associated reorganization), we need to update (and add) several namespaces, in particular the scheduler and executor APIs. These changes will allow the scheduler and executor APIs to better fit into the whole (in terms of mechanics, that is).

If we use the new approach, tasks would include:

  • Add meson.client.impl.master.scheduler
  • Add meson.client.impl.agent.executor
  • Remove meson.client.ClientAPI
  • Remove meson.client.Client
  • Remove SchedulerAPI
  • Remove SchedulerClient
  • Clean up meson.client
  • Move meson.client.base to meson.client.common and consolidate code

The following are partially done, but will be consistently added to over time:

  • Add convenience functions in meson.scheduler namespace
  • Add convenience functions in meson.executor namespace

Part of epic #2

Implement Master API

Functions required:

  • Master Protocol
    • bring-down-machines
    • bring-up-machines
    • create-volumes
    • destroy-volumes
    • get-agents
    • get-frameworks
    • get-maintenance-schedule
    • get-maintenance-status
    • get-quota
    • get-registry
    • get-roles
    • get-state-summary
    • get-tasks
    • redirect
    • remove-quota
    • reserve
    • set-quota
    • teardown-framework
    • unreserve
    • update-maintenance-scheduler
    • update-role-weights
  • Config Protocol
    • get-version
    • get-flags
  • Debug Protocol
    • start-profiler
    • stop-profiler
  • Files Protocol
    • browse
    • debug
    • download
    • read
  • Health Protocol
    • get-health
    • get-metrics
    • get-state
    • get-system-stats

Part of epic #2

Improved inheritance of BaseClient?

It would be nice if there was a nice way to simultaneously use and override an extended protocol/record ...

See BaseClient and SchedulerClientAPI for more context.

Deliver Clojure Client for v1.0 Mesos HTTP APIs

Milestone 0.1.0

  • Setup testing infrastructure - ticket #40
  • Add project documentation - ticket #65
  • Reorganize existing protobufs into modular namespaces - ticket #29
  • Add HTTP verb wrappers- ticket #24
  • Update namespaces for Meson Client, Scheduler, and Executor APIs - ticket #49
  • Support API event streams - ticket #56
  • Implement v1.0 Scheduler API - ticket #4

Milestone 0.2.0

  • Implement Agent API - ticket #59
  • Implement v1.0 Executor API - ticket #5

Milestone 0.3.0

  • Implement Master API - ticket #58
  • Implement v1.0 Operator API - ticket #6

Milestone 1.0.0

  • Add Meson Examples - ticket #73

Implement Operator API protobuf types

The operator API protobuf types were not part of Mesomatic, so these will have to be implemented from scratch.

This ticket depends upon the types namespace reorg ticket: #29

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.