Code Monkey home page Code Monkey logo

Comments (7)

GijsvanDulmen avatar GijsvanDulmen commented on July 4, 2024

@caiocampoos You can get a persistent cache setup using workspaces (https://tekton.dev/docs/pipelines/workspaces/) and a good enough Kubernetes Storage class for exampe.

from pipeline.

caiocampoos avatar caiocampoos commented on July 4, 2024

@GijsvanDulmen Reading the docs my understandig was with workspaces where always wiped when a pipeline finishes.

from pipeline.

caiocampoos avatar caiocampoos commented on July 4, 2024

@GijsvanDulmen you are correct. Ill try workspaces. Thanks.

from pipeline.

afrittoli avatar afrittoli commented on July 4, 2024

Thanks, @caiocampoos and @GijsvanDulmen. Workspaces are indeed an option for this.

Tekton does not provide any inbuilt mechanism for caches specifically, it's something that the Tasks that create and consume the cache have to manage directly through reusable workspaces. We do provide support for optional workspaces so that you can write a Task that can benefit from a cache if it's available but also use it when the cache workspace is not available.

The newly introduced StepActions are a good option to define reusable steps that may produce and restore a cache for a specific tool - I think this would be great additions to the Tekton catalog.

I'd be curious to hear about your experience with this, please let us know if you feel that Tekton could/should do more in this direction.

from pipeline.

caiocampoos avatar caiocampoos commented on July 4, 2024

@afrittoli thanks alot for the reply, i am having a hard time understanding the use of workspaces with persistentVolumeClaim at the moment.

Our usecase:

We have a pipeline of about 10k tests, so we need cache for dependencies and for Jest, witch is our test runner.

My current last try was:

apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerTemplate
metadata:
  name: tt-github-pr-trigger-template-
spec:
  params:
    - name: revision
    - name: deploy
    - name: repo-url
    - name: author
    - name: ref
    - name: repo-full-name
    - name: pr-ref
  resourcetemplates:
    - apiVersion: tekton.dev/v1beta1
      kind: PipelineRun
      metadata:
        generateName: pr-$(tt.params.pr-ref)-$(tt.params.author)-
      spec:
        serviceAccountName: service-account-
        pipelineRef:
          name: pipeline
        podTemplate:
          securityContext:
            fsGroup: 65532 
        workspaces:
          - name: shared-data
            persistentVolumeClaim:
              claimName: pvc-cache
        params:
          - name: repo-url
            value: $(tt.params.repo-url)
          - name: revision
            value: $(tt.params.revision)
          - name: repo-full-name
            value: $(tt.params.repo-full-name)
          - name: ref
            value: $(tt.params.ref)
          - name: deploy
            value: $(tt.params.deploy)

pipeline:
(i omit a good chunk just for sake of simplicity')

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: my-pipeline
spec:
  workspaces:
    - name: shared-data
  params:
    - name: repo-url
      type: string
    - name: revision
      type: string
    - name: repo-full-name
      type: string
    - name: ref
      type: string
    - name: deploy
      type: string
  tasks:
    - name: fetch-source
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: task-git-clone
          - name: namespace
            value: tekton-pipelines
      params:
        - name: url
          value: $(params.repo-url)
        - name: revision
          value: $(params.revision)
        - name: depth
          value: 2
      workspaces:
        - name: output
          workspace: shared-data
    - name: install-deps
      runAfter: ["update-status-running"]
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: task-install-deps
          - name: namespace
            value: tekton-pipelines
      params: 
        - name: install-script
          value: {{ .Values.install_script }}
        - name: post-install-script
          value: {{ default "echo no script" .Values.post_install_script }}
      workspaces:
        - name: source
          workspace: shared-data

Task: git-clone from docs

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: git-cli
  labels:
    app.kubernetes.io/version: "0.4"
  annotations:
    tekton.dev/pipelines.minVersion: "0.21.0"
    tekton.dev/categories: Git
    tekton.dev/tags: git
    tekton.dev/displayName: "git cli"
    tekton.dev/platforms: "linux/amd64,linux/s390x,linux/ppc64le"
spec:
  description: >-
    This task can be used to perform git operations.

    Git command that needs to be run can be passed as a script to
    the task. This task needs authentication to git in order to push
    after the git operation.

  workspaces:
    - name: source
      description: A workspace that contains the fetched git repository.

    - name: input
      optional: true
      description: |
        An optional workspace that contains the files that need to be added to git. You can
        access the workspace from your script using `$(workspaces.input.path)`, for instance:

          cp $(workspaces.input.path)/file_that_i_want .
          git add file_that_i_want
          # etc

    - name: ssh-directory
      optional: true
      description: |
        A .ssh directory with private key, known_hosts, config, etc. Copied to
        the user's home before git commands are executed. Used to authenticate
        with the git remote when performing the clone. Binding a Secret to this
        Workspace is strongly recommended over other volume types.

    - name: basic-auth
      optional: true
      description: |
        A Workspace containing a .gitconfig and .git-credentials file. These
        will be copied to the user's home before any git commands are run. Any
        other files in this Workspace are ignored. It is strongly recommended
        to use ssh-directory over basic-auth whenever possible and to bind a
        Secret to this Workspace over other volume types.
  params:
    - name: BASE_IMAGE
      description: |
        The base image for the task.
      type: string
      # TODO: Deprecate use of root image.
      default: cgr.dev/chainguard/git:root-2.39@sha256:7759f87050dd8bacabe61354d75ccd7f864d6b6f8ec42697db7159eccd491139

    - name: GIT_USER_NAME
      type: string
      description: |
        Git user name for performing git operation.
      default: ""

    - name: GIT_USER_EMAIL
      type: string
      description: |
        Git user email for performing git operation.
      default: ""

    - name: GIT_SCRIPT
      description: The git script to run.
      type: string
      default: |
        git help

    - name: USER_HOME
      description: |
        Absolute path to the user's home directory. Set this explicitly if you are running the image as a non-root user or have overridden
        the gitInitImage param with an image containing custom user configuration.
      type: string
      default: "/root"

    - name: VERBOSE
      description: Log the commands that are executed during `git-clone`'s operation.
      type: string
      default: "true"

  results:
    - name: commit
      description: The precise commit SHA after the git operation.

  steps:
    - name: git
      image: $(params.BASE_IMAGE)
      workingDir: $(workspaces.source.path)
      env:
      - name: HOME
        value: $(params.USER_HOME)
      - name: PARAM_VERBOSE
        value: $(params.VERBOSE)
      - name: PARAM_USER_HOME
        value: $(params.USER_HOME)
      - name: WORKSPACE_OUTPUT_PATH
        value: $(workspaces.output.path)
      - name: WORKSPACE_SSH_DIRECTORY_BOUND
        value: $(workspaces.ssh-directory.bound)
      - name: WORKSPACE_SSH_DIRECTORY_PATH
        value: $(workspaces.ssh-directory.path)
      - name: WORKSPACE_BASIC_AUTH_DIRECTORY_BOUND
        value: $(workspaces.basic-auth.bound)
      - name: WORKSPACE_BASIC_AUTH_DIRECTORY_PATH
        value: $(workspaces.basic-auth.path)
      script: |
        #!/usr/bin/env sh
        set -eu

        if [ "${PARAM_VERBOSE}" = "true" ] ; then
          set -x
        fi

        if [ "${WORKSPACE_BASIC_AUTH_DIRECTORY_BOUND}" = "true" ] ; then
          cp "${WORKSPACE_BASIC_AUTH_DIRECTORY_PATH}/.git-credentials" "${PARAM_USER_HOME}/.git-credentials"
          cp "${WORKSPACE_BASIC_AUTH_DIRECTORY_PATH}/.gitconfig" "${PARAM_USER_HOME}/.gitconfig"
          chmod 400 "${PARAM_USER_HOME}/.git-credentials"
          chmod 400 "${PARAM_USER_HOME}/.gitconfig"
        fi

        if [ "${WORKSPACE_SSH_DIRECTORY_BOUND}" = "true" ] ; then
          cp -R "${WORKSPACE_SSH_DIRECTORY_PATH}" "${PARAM_USER_HOME}"/.ssh
          chmod 700 "${PARAM_USER_HOME}"/.ssh
          chmod -R 400 "${PARAM_USER_HOME}"/.ssh/*
        fi

        # Setting up the config for the git.
        git config --global user.email "$(params.GIT_USER_EMAIL)"
        git config --global user.name "$(params.GIT_USER_NAME)"

        eval '$(params.GIT_SCRIPT)'

        RESULT_SHA="$(git rev-parse HEAD | tr -d '\n')"
        EXIT_CODE="$?"
        if [ "$EXIT_CODE" != 0 ]
        then
          exit $EXIT_CODE
        fi
        # Make sure we don't add a trailing newline to the result!
        printf "%s" "$RESULT_SHA" > "$(results.commit.path)"

I manage to use git clone task and aws cli using default secrets just fine when i mount my workspace as a:

   workspaces:
          - name: shared-data
            volumeClaimTemplate:
              spec:
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: 10Gi

But when i try to setup a pv and pvc to pass it as persistentVolumeClaim, i cannot manage to access those credentials anymore.

We are migrating our pipeline from CircleCi and i am very new to Tekton, so i would really appreciate a help here. Just from docs and examples i couldn't manage to figure workspaces for persistent cache out very well.

from pipeline.

caiocampoos avatar caiocampoos commented on July 4, 2024

I wanna add, that in previous tests i was able to persist data across pipeline runs and share between tasks, i am just confuse about how to setup credentials in the case of git or other tasks that use it.

from pipeline.

vdemeester avatar vdemeester commented on July 4, 2024

@caiocampoos note that you can have multiple workspaces, one with volumeClaimTemplate (and thus getting deleted when the PipelineRun is deleted) for the sources, and another one (backed by a pvc and peristentVolumeClaim) for the cache.

from pipeline.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.