Code Monkey home page Code Monkey logo

pyicemon's Introduction

pyicemon

Web front-end and framework for monitoring an icecream distributed build cluster. See icecc/icecream.

Usage

Basic usage:

python monitor.py <scheduler_hostname> <scheduler_port>

Then open visage.html in your browser.

Dependencies

websocket-server - pip install websocket-server

Status

Icemon API: All messages implemented as protocol version 22. Protocol version negotiation not supported.

Generic monitor: Fully implemented with generic publisher infrastructure.

Websocket Publisher: Implemented to send a json representation of the cluster down any connected websockets.

Web front-end: Not currently hosted by the monitor/publisher itself. But otherwise fundamentally works. Displays the cluster activity using d3.js force graph.

Class Design

Message (and subclasses): Represent a single message from the scheduler. They each know how to decode themselves from wire-data.

Usage:

from pyicemon import messages

# Unpack a message from a byte string. (string must contain only exactly one message)
m = messages.unpack(wire_data)

# Pack message to a byte string.
data = m.pack()

Connection: Handles a persistent connection to a scheduler. Reads messages off the wire and returns them as parsed Message types.

Usage:

import pyicemon
from pyicemon import messages

scheduler_host, scheduler_port = "scheduler.com", 8765

# Open a connection to a server. (Also handles version negotiation)
conn = pyicemon.Connection(scheduler_host, scheduler_port)

# Send a message.
m = messages.LoginMessage()
conn.send_message(m)

# Receive messages.
while True:
    m = conn.get_message()
    print m

Monitor: Maintains a view of the current state of the cluster by reading messages from a Connection and updating itself accordingly. Reports changes in cluster state to a Publisher.

Usage:

import pyicemon
from pyicemon.publishers import WebsocketPublisher

scheduler_host, scheduler_port = "scheduler.com", 8765
mon = pyicemon.Monitor(scheduler_host, scheduler_port)
mon.addPublisher(WebsocketPublisher(port=9999))
mon.run() # Blocks forever.

Publisher: Publishes information about the cluster to an outside source.

TODO

  • Implement protocol negotiation and interoperability between different protocol versions.
  • Generally improve quality of web view.
  • Automatic scheduler discovery.

pyicemon's People

Contributors

rynorris avatar

Watchers

 avatar  avatar

pyicemon's Issues

Sometimes nodes end up at negative load

Sometimes nodes can end up permanently stuck at negative active_jobs in the monitor.

It is unclear how this happens. Presumably we get a JobDone message without having received the corresponding JobBegin, so the count of active_jobs is off.

Potential fix is to actually maintain a list of job IDs the CS is doing, and then use the length of that list as the number of active jobs instead of trying to keep count manually.

We don't support proper protocol negotiation.

In fact, we assert and crash if the server doesn't respond with version 22.

We should be able to handle any protocol version, the conversation between scheduler and monitor hasn't changed much.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.