Code Monkey home page Code Monkey logo

go-dexec's Introduction

dexec GoDoc

dexec is a small Go library allowing you to run processes inside Docker containers as if you are running them locally using os/exec package. Read documentation at godoc.org or see examples.

Using dexec, you can do stream processing, off-load computationally expensive parts of your program to a remote fleet of Docker engines. Its interface is strikingly similar to os/exec.

Examples

Check out the following examples:

A Short Example

It takes only a 4-line code change to convert a piece of code using os/exec to use dexec to start running your stuff inside containers.

Here is a minimal Go program that runs echo in a container:

package main

import (
	"log"

	"github.com/ahmetalpbalkan/dexec"
	"github.com/fsouza/go-dockerclient"
)

func main(){
	cl, _ := docker.NewClient("unix:///var/run/docker.sock")
	d := dexec.Docker{cl}

	m, _ := dexec.ByCreatingContainer(docker.CreateContainerOptions{
	Config: &docker.Config{Image: "busybox"}})

	cmd := d.Command(m, "echo", `I am running inside a container!`)
	b, err := cmd.Output()
	if err != nil { log.Fatal(err) }
	log.Printf("%s", b)
}

Output: I am running inside a container!

Use Cases

This library is intended for providing an execution model that looks and feels like os/exec package.

You might want to use this library when:

  • You want to execute a process, but run it in a container with extra security and finer control over resource usage with Docker –and change your code minimally.

  • You want to execute a piece of work on a remote machine (or even better, a pool of machines or a cluster) through Docker. Especially useful to distribute computationally expensive workloads.

For such cases, this library abstracts out the details of executing the process in a container and gives you a cleaner interface you are already familiar with.

Check out more examples →

Read more →

Analytics

go-dexec's People

Contributors

ahmetb avatar sheenobu avatar trp-samer-adra avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-dexec's Issues

stdin stdout issue

I am having an issue with getting stdin and stdout to work properly at the same time. Hopefully the snippet makes sense. I'm trying to read in a file and process it as stdin to the container and then return the result back to an http responseWriter. There are two issues here. It will work and process the data and return to os.Stdout if I set that... sending cmd.Stdout to a byte buffer never happens as far as I can tell... also the container stays running in the background until I close the program... It seems like its blocking for some reason.

func toUpperDocker(w http.ResponseWriter, filename string) {
    client := clientConn()
    m, err := dexec.ByCreatingContainer(docker.CreateContainerOptions{
        Config: &docker.Config{Image: "busybox"}})
    d := dexec.Docker{client}
    cmd := d.Command(m, "tr", "[:lower:]", "[:upper:]")
    log.Println(cmd)

    log.Printf("Read file %s", filename)
    file, err := ioutil.ReadFile(filename)
    if err != nil {
        log.Printf("ReadFile error: %v\n", err)
    }

    log.Println("Starting StdinPipe")
    wc, err := cmd.StdinPipe()
    if err != nil {
        log.Printf("StdinPipe error: %s\n", err)
    }
    log.Println("Starting StdoutPipe")
    rc, err := cmd.StdoutPipe()
    if err != nil {
        log.Printf("StdoutPipe error: %s\n", err)
    }
    var out bytes.Buffer
    // cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stdout
    cmd.Stdout = &out
    err = cmd.Start()
    if err != nil {
        log.Fatal(err)
    }

    //stream the file to the WriteCloser
    log.Printf("Attempting to stream %s to writecloser %v bytes to send.", filename, len(file))
    n, err := fmt.Fprint(wc, string(file))
    fmt.Fprintln(wc, "EOF")
    if err != nil {
        log.Printf("file Fprint error: %s\n", err)
    }
    log.Printf("%d bytes streamed to the writeCloser", n)

    if err := cmd.Wait(); err != nil {
        log.Fatal(err)
    }

    log.Println("close writecloser")
    err = wc.Close()
    if err != nil {
        log.Printf("WriteCloser error: %s\n", err)
    }

    // c1 := exec.Command("ls")
    // c1.Stdout = &out
    // err = c1.Start()
    // if err != nil {
    //  log.Fatal(err)
    // }

    n64, err := out.ReadFrom(rc)
    if err != nil {
        log.Fatal(err)
    }
    log.Printf("out.readfrom number of bytes: %v", n64)

    err = rc.Close()
    if err != nil {
        log.Printf("ReadCloser error: %s\n", err)
    }
    w.WriteHeader(http.StatusOK)
    w.Write(out.Bytes())

}

Ability to run commands inside an existing container?

Might be missing something, but from a cursory glance it doesn't appear to be possible to run a command inside an existing container?

My usecase is I'd like to be able to call into a container which runs some caching daemon, and the commands are passed into the daemon running in the container via a in-container cli tool.

Missing output from fast-running commands

I wanna run sh -c "echo $(pwd)".

cmd is executed by dexec, but it doesn't output anything.
cmd2 is executed by go's built-in exec , it works.

package main

import (
  "log"

  "github.com/ahmetalpbalkan/dexec"
  "github.com/fsouza/go-dockerclient"
  "os/exec"
)

func main() {
  cl, _ := docker.NewClient("unix:///var/run/docker.sock")
  d := dexec.Docker{cl}

  m, _ := dexec.ByCreatingContainer(docker.CreateContainerOptions{
    Config: &docker.Config{Image: "busybox"}})

  cmd := d.Command(m, "sh", "-c", "echo $(pwd)")
  b, err := cmd.Output()
  if err != nil {
    log.Fatal(err)
  }
  log.Printf("%s", b)

  cmd2 := exec.Command("sh", "-c", "echo $(pwd)")
  bb, err := cmd2.Output()
  if err != nil {
    log.Fatal(err)
  }
  log.Printf("%s", bb) 

}


docker volume parameter

Hi, nice project.
Is it possible to share a volume and files using your library (ex: docker run -v /home/user/:/home/ container ) ?

Unexpected behavior caused by obsolete imports in tests

I have a project that depends on this dependency and uses go modules and runs go mod tidy.
About five months ago the pipeline started failing despite no change no my end.
What was happening was a test that was timing out when running a Docker command using this package.
I noticed that go mod tidy kept adding an indirect dependency to "github.com/ahmetalpbalkan/go-dexec" and when that dependency was in my go.mod (along with the direct dependency on this lib in its current location under ahmetb), the test would timeout. When the "github.com/ahmetalpbalkan/go-dexec" indirect dependency was not in go.mod, the test succeeded.

go mod why github.com/ahmetalpbalkan/go-dexec revealed the indirect dependency was getting pulled in because the old package is still being referenced by several tests (and go mod tidy isn't smart enough to exclude that).

Simply need to change ahmetalpbalkan to ahmetb as the old repo and its packages no longer exist.

undefined: docker.CloseWaiter

trying to run the tests and getting

$ go test
# github.com/ahmetalpbalkan/go-dexec
./execution.go:26: undefined: docker.CloseWaiter

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.