Comments (7)
@caiocampoos You can get a persistent cache setup using workspaces (https://tekton.dev/docs/pipelines/workspaces/) and a good enough Kubernetes Storage class for exampe.
from pipeline.
@GijsvanDulmen Reading the docs my understandig was with workspaces where always wiped when a pipeline finishes.
from pipeline.
@GijsvanDulmen you are correct. Ill try workspaces. Thanks.
from pipeline.
Thanks, @caiocampoos and @GijsvanDulmen. Workspaces are indeed an option for this.
Tekton does not provide any inbuilt mechanism for caches specifically, it's something that the Tasks
that create and consume the cache have to manage directly through reusable workspaces. We do provide support for optional workspaces so that you can write a Task
that can benefit from a cache if it's available but also use it when the cache workspace is not available.
The newly introduced StepActions
are a good option to define reusable steps that may produce and restore a cache for a specific tool - I think this would be great additions to the Tekton catalog.
I'd be curious to hear about your experience with this, please let us know if you feel that Tekton could/should do more in this direction.
from pipeline.
@afrittoli thanks alot for the reply, i am having a hard time understanding the use of workspaces with persistentVolumeClaim at the moment.
Our usecase:
We have a pipeline of about 10k tests, so we need cache for dependencies and for Jest, witch is our test runner.
My current last try was:
apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerTemplate
metadata:
name: tt-github-pr-trigger-template-
spec:
params:
- name: revision
- name: deploy
- name: repo-url
- name: author
- name: ref
- name: repo-full-name
- name: pr-ref
resourcetemplates:
- apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: pr-$(tt.params.pr-ref)-$(tt.params.author)-
spec:
serviceAccountName: service-account-
pipelineRef:
name: pipeline
podTemplate:
securityContext:
fsGroup: 65532
workspaces:
- name: shared-data
persistentVolumeClaim:
claimName: pvc-cache
params:
- name: repo-url
value: $(tt.params.repo-url)
- name: revision
value: $(tt.params.revision)
- name: repo-full-name
value: $(tt.params.repo-full-name)
- name: ref
value: $(tt.params.ref)
- name: deploy
value: $(tt.params.deploy)
pipeline:
(i omit a good chunk just for sake of simplicity')
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: my-pipeline
spec:
workspaces:
- name: shared-data
params:
- name: repo-url
type: string
- name: revision
type: string
- name: repo-full-name
type: string
- name: ref
type: string
- name: deploy
type: string
tasks:
- name: fetch-source
taskRef:
resolver: cluster
params:
- name: kind
value: task
- name: name
value: task-git-clone
- name: namespace
value: tekton-pipelines
params:
- name: url
value: $(params.repo-url)
- name: revision
value: $(params.revision)
- name: depth
value: 2
workspaces:
- name: output
workspace: shared-data
- name: install-deps
runAfter: ["update-status-running"]
taskRef:
resolver: cluster
params:
- name: kind
value: task
- name: name
value: task-install-deps
- name: namespace
value: tekton-pipelines
params:
- name: install-script
value: {{ .Values.install_script }}
- name: post-install-script
value: {{ default "echo no script" .Values.post_install_script }}
workspaces:
- name: source
workspace: shared-data
Task: git-clone from docs
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: git-cli
labels:
app.kubernetes.io/version: "0.4"
annotations:
tekton.dev/pipelines.minVersion: "0.21.0"
tekton.dev/categories: Git
tekton.dev/tags: git
tekton.dev/displayName: "git cli"
tekton.dev/platforms: "linux/amd64,linux/s390x,linux/ppc64le"
spec:
description: >-
This task can be used to perform git operations.
Git command that needs to be run can be passed as a script to
the task. This task needs authentication to git in order to push
after the git operation.
workspaces:
- name: source
description: A workspace that contains the fetched git repository.
- name: input
optional: true
description: |
An optional workspace that contains the files that need to be added to git. You can
access the workspace from your script using `$(workspaces.input.path)`, for instance:
cp $(workspaces.input.path)/file_that_i_want .
git add file_that_i_want
# etc
- name: ssh-directory
optional: true
description: |
A .ssh directory with private key, known_hosts, config, etc. Copied to
the user's home before git commands are executed. Used to authenticate
with the git remote when performing the clone. Binding a Secret to this
Workspace is strongly recommended over other volume types.
- name: basic-auth
optional: true
description: |
A Workspace containing a .gitconfig and .git-credentials file. These
will be copied to the user's home before any git commands are run. Any
other files in this Workspace are ignored. It is strongly recommended
to use ssh-directory over basic-auth whenever possible and to bind a
Secret to this Workspace over other volume types.
params:
- name: BASE_IMAGE
description: |
The base image for the task.
type: string
# TODO: Deprecate use of root image.
default: cgr.dev/chainguard/git:root-2.39@sha256:7759f87050dd8bacabe61354d75ccd7f864d6b6f8ec42697db7159eccd491139
- name: GIT_USER_NAME
type: string
description: |
Git user name for performing git operation.
default: ""
- name: GIT_USER_EMAIL
type: string
description: |
Git user email for performing git operation.
default: ""
- name: GIT_SCRIPT
description: The git script to run.
type: string
default: |
git help
- name: USER_HOME
description: |
Absolute path to the user's home directory. Set this explicitly if you are running the image as a non-root user or have overridden
the gitInitImage param with an image containing custom user configuration.
type: string
default: "/root"
- name: VERBOSE
description: Log the commands that are executed during `git-clone`'s operation.
type: string
default: "true"
results:
- name: commit
description: The precise commit SHA after the git operation.
steps:
- name: git
image: $(params.BASE_IMAGE)
workingDir: $(workspaces.source.path)
env:
- name: HOME
value: $(params.USER_HOME)
- name: PARAM_VERBOSE
value: $(params.VERBOSE)
- name: PARAM_USER_HOME
value: $(params.USER_HOME)
- name: WORKSPACE_OUTPUT_PATH
value: $(workspaces.output.path)
- name: WORKSPACE_SSH_DIRECTORY_BOUND
value: $(workspaces.ssh-directory.bound)
- name: WORKSPACE_SSH_DIRECTORY_PATH
value: $(workspaces.ssh-directory.path)
- name: WORKSPACE_BASIC_AUTH_DIRECTORY_BOUND
value: $(workspaces.basic-auth.bound)
- name: WORKSPACE_BASIC_AUTH_DIRECTORY_PATH
value: $(workspaces.basic-auth.path)
script: |
#!/usr/bin/env sh
set -eu
if [ "${PARAM_VERBOSE}" = "true" ] ; then
set -x
fi
if [ "${WORKSPACE_BASIC_AUTH_DIRECTORY_BOUND}" = "true" ] ; then
cp "${WORKSPACE_BASIC_AUTH_DIRECTORY_PATH}/.git-credentials" "${PARAM_USER_HOME}/.git-credentials"
cp "${WORKSPACE_BASIC_AUTH_DIRECTORY_PATH}/.gitconfig" "${PARAM_USER_HOME}/.gitconfig"
chmod 400 "${PARAM_USER_HOME}/.git-credentials"
chmod 400 "${PARAM_USER_HOME}/.gitconfig"
fi
if [ "${WORKSPACE_SSH_DIRECTORY_BOUND}" = "true" ] ; then
cp -R "${WORKSPACE_SSH_DIRECTORY_PATH}" "${PARAM_USER_HOME}"/.ssh
chmod 700 "${PARAM_USER_HOME}"/.ssh
chmod -R 400 "${PARAM_USER_HOME}"/.ssh/*
fi
# Setting up the config for the git.
git config --global user.email "$(params.GIT_USER_EMAIL)"
git config --global user.name "$(params.GIT_USER_NAME)"
eval '$(params.GIT_SCRIPT)'
RESULT_SHA="$(git rev-parse HEAD | tr -d '\n')"
EXIT_CODE="$?"
if [ "$EXIT_CODE" != 0 ]
then
exit $EXIT_CODE
fi
# Make sure we don't add a trailing newline to the result!
printf "%s" "$RESULT_SHA" > "$(results.commit.path)"
I manage to use git clone task and aws cli using default secrets just fine when i mount my workspace as a:
workspaces:
- name: shared-data
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
But when i try to setup a pv and pvc to pass it as persistentVolumeClaim, i cannot manage to access those credentials anymore.
We are migrating our pipeline from CircleCi and i am very new to Tekton, so i would really appreciate a help here. Just from docs and examples i couldn't manage to figure workspaces for persistent cache out very well.
from pipeline.
I wanna add, that in previous tests i was able to persist data across pipeline runs and share between tasks, i am just confuse about how to setup credentials in the case of git or other tasks that use it.
from pipeline.
@caiocampoos note that you can have multiple workspaces, one with volumeClaimTemplate
(and thus getting deleted when the PipelineRun is deleted) for the sources, and another one (backed by a pvc and peristentVolumeClaim) for the cache.
from pipeline.
Related Issues (20)
- Add the ability to override the `timeout` value for individual pipeline tasks via taskRunSpecs in a pipelineRun
- The params in step are passed to `stdoutConfig.path`. HOT 1
- [StepAction] taskSpec.results in Pipeline not works with workspaces HOT 5
- Tekton pipeline validation crashes on script validation in Openshift HOT 1
- Tekton upgrade to v0.56.2 is broken HOT 7
- TaskRun Fails to Create Pod When ServiceAccount References Non-Existent Secret HOT 3
- Unexpected Default βkindβ in taskRef When Using Resolver in Tekton Pipelines HOT 1
- multiple interpolations in stepResult substitution should be allowed HOT 1
- Pipeline fails with default array params when type is not specified in remote resource HOT 1
- Action Button not visible HOT 2
- Is the `retries` count for Task without validation by design?
- Support full podtemplate spec for affinity assistant pod HOT 1
- When debug is not turned on correctly, redundant volumes appear HOT 1
- Add support for integer in params of pipeline
- Timeout not working on Pipeline run HOT 4
- [0.56.x] stepaction isn't configured for conversion webhook HOT 1
- Workingdir does not give write access to the group HOT 2
- TEP-0142 Promote StepActions to Beta HOT 3
- WIP: TaskRun recoverable error, `CreateContainerConfigError` neither resumes PipelineRun on fix, nor times out
- Pipelinerun and Taskrun Count metrics should have less cardinality HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pipeline.