Code Monkey home page Code Monkey logo

kubejob's Introduction

kubejob

PkgGoDev Go codecov

A library for managing Kubernetes Job in Go

Features

  • Creates a Kubernetes Job
  • Run and wait for Kubernetes Job
  • Capture logs of Kubernetes Job
  • Can use context.Context to run Kubernetes Job
  • Delayed execution of Kubernetes Job ( also support Sidecar pattern )
  • Can control execution order ( and timing ) of command for multiple containers at Job
  • Copy any files or directory between local file system and container in Job
  • Can insert any process before the process of the init container
  • Automatically clean up Kubernetes Job

Installation

$ go get github.com/goccy/kubejob

Synopsis

Simply create and run and wait Kubernetes Job

import (
  "github.com/goccy/kubejob"
  batchv1 "k8s.io/api/batch/v1"
  corev1 "k8s.io/api/core/v1"
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
  "k8s.io/client-go/rest"
)

var cfg *rest.Config // = rest.InClusterConfig() assign *rest.Config value to cfg

job, err := kubejob.NewJobBuilder(cfg, "default").BuildWithJob(&batchv1.Job{
  ObjectMeta: metav1.ObjectMeta{
    GenerateName: "kubejob-",
  },
  Spec: batchv1.JobSpec{
    Template: corev1.PodTemplateSpec{
      Spec: corev1.PodSpec{
        Containers: []corev1.Container{
          {
            Name:    "test",
            Image:   "golang:1.21.0-bookworm",
            Command: []string{"echo", "hello"},
          },
        },
      },
    },
  },
})
if err != nil {
  panic(err)
}

// Run KubernetesJob and output log to stdout.
// If Job doesn't exit status by `0` , `Run()` returns error.
if err := job.Run(context.Background()); err != nil {
	panic(err)
}

Manage execution timing

If you don't want to execute the Job immediately after the Pod is Running state, you can delay the execution timing. Also, this feature can be used if the Job has multiple containers and you want to control their execution order.

ctx := context.Background()
if err := job.RunWithExecutionHandler(ctx, func(executors []*kubejob.JobExecutor) error {
  // callback when the Pod is in Running status.
  // `executors` corresponds to the list of `Containers` specified in `JobSpec`
  for _, exec := range executors { 
    // `Exec()` executes the command
    out, err := exec.Exec()
    if err != nil {
      panic(err)
    }
    fmt.Println(string(out), err)
  }
  return nil
})

Manage execution with Sidecar

job, err := kubejob.NewJobBuilder(cfg, "default").BuildWithJob(&batchv1.Job{
  ObjectMeta: metav1.ObjectMeta{
    GenerateName: "kubejob-",
  },
  Spec: batchv1.JobSpec{
    Template: corev1.PodTemplateSpec{
      Spec: corev1.PodSpec{
        Containers: []corev1.Container{
          {
            Name:    "main",
            Image:   "golang:1.21.0-bookworm",
            Command: []string{"echo", "hello"},
          },
          {
            Name:    "sidecar",
            Image:   "nginx:latest",
            Command: []string{"nginx"},
          },
        },
      },
    },
  },
})
if err != nil {
  panic(err)
}
if err := job.RunWithExecutionHandler(context.Background(), func(executors []*kubejob.JobExecutor) error {
  for _, exec := range executors {
    if exec.Container.Name == "sidecar" {
      // `ExecAsync()` executes the command and doesn't wait finished.
      exec.ExecAsync()
    } else {
      out, err := exec.Exec()
      if err != nil {
        panic(err)
      }
      fmt.Pritln(string(out))
    }
  }
  return nil
})

Execution with kubejob-agent

Normally, copying and executing commands using JobExecutor is performed using the shell or tar command in the container image and using the Kubernetes API ( pods/exec ). However, if you need to copy large files or run a large number of commands and don't want to overload the Kubernetes API, or if you don't have a shell or tar command in your container image, you can use the kubejob-agent method.

kubejob-agent is a CLI tool that can be installed by the following methods.

$ go install github.com/goccy/kubejob/cmd/kubejob-agent

Include this tool in your container image, create kubejob.AgentConfig with the path where kubejob-agent is located, and give it to kubejob.Job as follows:

agentConfig, err := kubejob.NewAgentConfig(map[string]string{
  "container-name": filepath.Join("/", "bin", "kubejob-agent"),
})
if err != nil {
  return err
}
job.UseAgent(agentConfig)

This will switch from communication using the Kubernetes API to gRPC-based communication with kubejob-agent. Communication with kubejob-agent is performed using JWT issued using the RSA Key issued each time Kubernetes Job is started, so requests cannot be sent directly to the container from other processes.

Requirements

Role

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubejob
rules:
  - apiGroups:
      - batch
    resources:
      - jobs
    verbs:
      - create
      - delete
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
      - list
      - watch
      - delete
  - apiGroups:
      - ""
    resources:
      - pods/log
    verbs:
      - get
      - watch
  - apiGroups:
      - ""
    resources:
      - pods/exec
    verbs:
      - create # required when using kubejob.JobExecutor without kubejob-agent

Tools

cmd/kubejob

Installation

go install github.com/goccy/kubejob/cmd/kubejob
Usage:
  kubejob [OPTIONS]

Application Options:
  -n, --namespace= specify namespace (default: default)
  -f, --file=      specify yaml or json file for written job definition
  -i, --image=     specify container image

Help Options:
  -h, --help       Show this help message

Example

$ kubejob --image golang:1.21.0-bookworm -- go version
go version
go version go1.21.0 linux/amd64

kubejob's People

Contributors

ainoya avatar dependabot[bot] avatar gnagel avatar goccy avatar yutachaos avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

kubejob's Issues

How to not display pod logs?

I don't want to display the pod log in the main program(because it's too much), i set log level to LogLevelError, but still dispaly logs.
job.SetLogLevel(kubejob.LogLevelError)

Is there a method to prevent the main program from displaying the pod log?

job stopped after 4 hours

I started a long running task using kubejob(job.Run), but the job complete after 4hours, even if the pod(started by the job) is running. No error log output, it's like kubejob has a 4-hour timeout setting.

Is there a way to cancel or alter this setting?

Indexed Kube Job

Is is possible to run an indexed kube job?

When running a job with: completionIndex = 2, parallel = 1 or 2 the Run method returns after the first pod is completed.

Is it possible to block on all pods completed, or the job is failed?

Ty!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.