Code Monkey home page Code Monkey logo

gows's Introduction

Gows - A Golang Workflow Scheduler

Go

Gows is an easy and super lightweight workflow management tool. There is no need for setting up databases and web applications, Gows only takes a couples of json configuration files and that is it. It is all that it takes to define your workflow DAGs.

Dag configuration

Gows's dag are describe using a json file :

{
   "dagName":"example-2",
   "schedule":"*/3 * * * * *",
   "tasks":[
      {
         "name":"bash-task-1",
         "type":"bash",
         "parameters":{
            "cmd":"ls -ah"
         }
      },
      {
         "name":"bash-task-2",
         "type":"bash",
         "parameters":{
            "cmd":"echo 'toto'"
         }
      },
      {
         "name":"bash-task-3",
         "type":"bash",
         "parameters":{
            "cmd":"sleep 2"
         },
         "dependencies":[
            "bash-task-1",
            "bash-task-2"
         ]
      }
   ]
}

Package

Gows core package can be used without gows cli :

go get -u com.github/Fszta/Gows

dag, _ := CreateDag("dag-example")

operator1 := operators.CreateBashOperator()
operator1.SetCmd("ls -lah")
task1, _ := task.CreateTask(operator1, "task1")
dag.AddTask(task1)

operator2 := operators.CreateBashOperator()
operator2.SetCmd("sleep 1 && tree")
task2, _ := task.CreateTask(operator2, "task2")
dag.AddTask(task2)
dag.SetDependency(task2, task1)

operator3 := operators.CreateBashOperator()
task3, _ := task.CreateTask(operator3, "task3")
operator3.SetCmd("ps")
dag.AddTask(task3)
dag.SetDependency(task3, task2)

dag.Run()

gows's People

Contributors

fszta avatar yinshangyi avatar

Stargazers

 avatar

gows's Issues

Task implementation

Implement the task struct which should create an id for itself and instanciate executors.
Task is the factory for the executors.

Dag visualization

Add function to visualise dag in console :
.........................|[3]
[1]-----[2]-----|
.........................|[4]

Store dag in embedded db

Use boltd db to manage all added dag into db :

  • Retrieve all dag when init gows
  • Add new dag when gows dag add is called
  • Remove dag when gows dag remove is called

Dag configuration format

A proper JSON structured should be defined.
Different types of operator may have different types of configurations.
This problem should be addressed in this issue

JSON to Dag generator

This module should parse a JSON configuration file and generate a Dag object.
The JSON configuration should have the following format:

{
  "dagName": "hello",
  "schedule": "*/3 * * * * *",
  "operators": [
    {
      "name": "BashOperatorOne",
      "type": "bash",
      "parameters": {
        "cmd": "test.sh",
        "arguments": [
          {
            "arg": "-t",
            "value": "1"
          },
          {
            "arg": "-d",
            "value": "detached"
          }
        ]
      }
    },
    {
      "name": "BashOperatorTwo",
      "type": "bash",
      "parameters": {
        "cmd": "test.sh -t 1 -d detached"
      }
    },
    {
      "name": "PythonOperator1",
      "type": "python",
      "parameters": {
        "src": "test.py",
        "virtualenv": "venv",
        "arguments": [
          {
            "arg": "-t",
            "value": "1"
          }
        ]
      }
    },
    {
      "name": "PythonOperator1",
      "type": "python",
      "parameters": {
        "src": "test.py"
      }
    }
  ]
}

Using this solution, there is no fixed field.
Each operator have their own configuration fields, there is no uniformity among the operators in terms of configuration
fields since we don't use a factory pattern anymore?
This implies to have a strong fields validation process with something similar to this :

if (operator.type == "python") {
   for _, arg := range argsList {
      if (!isArgInConf(operator.type)) {
         return false
     }
   } 
   return true
} 

Another approach would be to have fixed field with a classic unmarshalling.

Given that we didn't use a factory pattern, perhaps the no fixed field approach make more sense.

Implement the Task

Implement the task struct which should create an id for itself and instanciate executors.
Task is the factory for the executors.

Implement the CLI user interface

Implement the CLI to support all the commands.
This is only about the user interaction, there is no need for integration with the core functions of Gows for this issue

Define CLI scenario

  • Brainstorm about the CLI possible commands/scenario
  • Writing the different scenarios
  • Possibly writing the documentation

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.