earonesty / grun Goto Github PK

* Description:

A lightweight replacement for job queueing systems like LSF, Torque, condor, SGE, for private clusters.

* Installation:

You need perl and ZMQ::LibZMQ3

    apt-get install libzmq3
    cpanm ZMQ::LibZMQ3

Put it in /usr/bin.   Type grun -h for help with the configuration/setup.  Basically you stick it on all the machines, and you put a "master" in the main config.  It is, far and away, the easiest grid job queuing system out there.

Example /etc/grun.conf on the master node:

master:    foo.mydomain.local
services:   queue
log_file:  /var/log/grun.log

Example /etc/grun.conf on a many compute nodes:

master:    foo.mydomain.local
services:   exec
log_file:  /var/log/grun.log

Once you have those two daemons running (grun -d), then you can, on either node, do this:

    grun hostname

And it will show you which node the command was run on, confirming that your installtion works.

* What you do with it:

Submit jobs, grun puts them in a queue, the queue is on disk so machines can lose connection any time, and jobs keep going.   Specify resouces requirements, or not - and have it figure stuff out.

grun will assign jobs randomly, and will tend to fill up one machine at a time, in case big jobs come along.   It should figure NFS mounts, and the right thing to do with them autoamtically.  It has a fast i/o subsystem suitable for terabyte files.  If all the machines are busy, jobs will queue up.  Resources are soft-locked by jobs until they are finished, and if a non-grun job is using a machine, it's accounted for. 

NOTE: Jobs that require changing sets of resources should call grun themselves - forcing later commands to requeue:

IE:

> grun -m 3000 myjob.sh

--- myjob.sh: ---
perl usesalotofmemory.pl
grun -c 4 -m 500 perl uses4cpusbutlessram.pl

The memory and cpu usage of the parent program will go to zero as the second script is launched.   In this way very complex jobs can be scheduled, without a special workflow system.  Just use bash, perl or ruby.

* What it does now:

It does the queueing, i/o, node-matching to requirements, suspend/resume, priorities, and has a great config system.   The "auto conf" allows sysadmins to create policies based on the parameters or script contens of a job.  Grun has run millions of jobs in a production cluster with about 52 nodes of 24-48 cores each.  Grun times every command and records memory/cpu/io.  Ulimits are set on jobs, and can be scaled to a multiple of requested (elbowing).   Jobs are editable, killable & suspend/resumable.  You can specify "make like" semantics on jobs and it will check inputs/outputs and run jobs only if needed.
 
Grun comes with a perl module that allows you to execute perl functions on remote hosts by copying the memory and code over to the remote machine. For example:

    use Grun;
    my $results = grun(\&mycrazyfunction, "param");

It decompiles the function locally, and recompiles remotely.

* What it doesn't do yet (TODO):

- needs a "plugin" architecture.  Should be easy with perl.
- graceful restart... with socket handoff (it's always safe to restart... but it can cause waiters to pause)
- harrass users when queue drive is close to full, or cpu/ram was too high to ever run, or other (configurable) issues that users have

* Goals:

Small           Keep the code under a few k lines, should be readable.   Lots of comments.
Simple          Easy configuration, guesses default values
Smart           Keeps track of stats.  Learns what to do with things.   Puts the right jobs on the right machines.
Fast            Hundreds of thousands of jobs per hour on tens of thousands of machines should be easy.
Configurable    Make up lots of new things you want it to keep track of and allocate to jobs (like disk i/o, user limits, etc.)

* Features to avoid:

No security     Grun has no security.   It has to be behind a firewall on dedicated machines.  This limits it, and keeps it simple.  It's not hard to put up an ssh tunnel, and make it work at EC2.   But I'm not building in kerberos or stuff like that.

One platform    Grun only works on unix-like machines that have perl installed.

Nothing fancy   Grun doesn't support MPI and other fancy grid things (although you can layer map-reduce on it)

* Advanced usage: 

Example of a config on an ssh tunneled EC2 node (tested), with autostaging via gwrap:

services:       exec
master:         127.0.0.1:5184
bind:           127.0.0.2:5184
wrap:           /usr/bin/gwrap
fetch_path:     /opt:/mnt/ufs

gwrap (source only) is a useful tool.  Used right, it will copy all the dependent files to the remote host, only as needed, and then, when the remote program is finished, it will copy all the output files back to the local host.   Of course, this *could* be built in to grun, via the -M option, but it isn't yet.   gwrap is not yet heavily used in production, and if you do decide to use it heavly, please let me know... I'll help add things like logging, failure recovery as needed.