daskos / mentos Goto Github PK

View Code? Open in Web Editor NEW

18.0 4.0 3.0 248 KB

Fresh Python Mesos Scheduler and Executor driver

License: Apache License 2.0

Python 99.43% Shell 0.57%

python mesos http client scheduler resource-manager

mentos's Introduction

Apache Mesos backend for Dask scheduling library. Run distributed python dask workflows on your Mesos cluster.

Notable Features

distributively run tasks in docker container
specify resource requirements per task
bin packing for optimized resource utilization

Installation

Prerequisits: satyr, dask, toolz. All of them should be installed w/ the following commands:

pip install dask.mesos or use lensa/dask.mesos Docker image

Configuration:

MESOS_MASTER=zk://127.0.0.1:2181/mesos
ZOOKEEPER_HOST=127.0.0.1:2181

Example

from __future__ import absolute_import, division, print_function

from daskos import mesos, MesosExecutor


@mesos(cpus=0.1, mem=64)
def add(x, y):
    """Run add on mesos with specified resources"""
    return x + y


@mesos(cpus=0.3, mem=128, image='lensa/daskos')
def mul(x, y):
    """Run mul on mesos in specified docker image"""
    return x * y


with MesosExecutor(name='dask') as executor:
    """This context handles Mesos scheduler's lifecycle"""
    a, b = 23, 89
    alot = add(a, b)
    gigalot = mul(alot, add(10, 2))

    gigalot.compute(get=executor.get)  # (a + b) * (10 + 2)
    executor.compute([alot, gigalot])  # list of futures

Configuring daskos Tasks

You can configure your mesos tasks in your decorator, currently the following options are available:

cpus: The amount of cpus to use for the task.
mem: Memory in MB to reserver for the task.
disk: The amount of disk to use for the task.
image: A docker image name. If not set, mesos containerizer will be used.

Both mem and cpus are defaults to some low values set in satyr so you're encouraged to override them here. More options like constraints, other resources are on the way.

mentos's People

Contributors

Stargazers

Watchers

Forkers

arttii kszucs fifan31

mentos's Issues

Adding Proper State Management akin to zoonado and aiozk

Adding proper state transitions to transparently understand what state the driver is in at the moment.

https://github.com/wglass/zoonado/blob/master/zoonado/states.py

Come to think of it, there are a bunch of patterns, that could be stolen from zoonado, to make the code a bit cleaner, specifically the reconnect and discovery semantics.

Explicit method names in debug messages

https://github.com/daskos/mentos/blob/master/mentos/interface.py

I think we should avoid using sys._get_frame()

Reconnect does not work properly with redirect master discovery

Adding a reconnect in the header handle of subscribe fixes this, but I dont like this.

Satyr integration

Satyr should be updated:

use mentos directly, without proxies
CI: drone -> compose + travis

Org Name

Issue tracking a new organization name.

@kszucs

Collaborator request

Can You add me as an outside collaborator?

I'd create a drone CI integration.

Rafactor to make the interface a bit less magical

Malefico isn't easily memorizable. It's not intuitive and I don't know the meaning of it (is it italian https://translate.google.com/#auto/en/malefico or a character?)
Satyr's name also odd, difficult to find.

I think malefico should be renamed to suggest "it's a native mesos client". Satyr should be renamed to suggest "it's a high level mesos framework". Ideas?

Do You have other names or you insist on malefico?

Tests

I ended up moving to travis and coveralls, because it was just simpler to set up. I duplicated the drone config via docker-compose. It seems to work fine for now.

Not all tests are currently there. I will list them out here:

Utils contains the zookeeper discovery, which I tested by killing one of the masters using docker-compose and waiting for some intervals in the middle. Seems like its working.

@kszucs Any thoughts?

Weird edgecase with redirect master Discovery

If zookeeper is turned off, but Mesos Master is running and we try to subscribe the redirect master detection get's an address, which when subscribed to, again redirects. The detector eventually reaches the actual Master, because currently there is no gate if there is already an active Master