Comments (10)
Hey Jason - thanks for bringing this up. There's a bunch of potential routes to take here and I could spin up a charter for the team to investigate but before doing so, I'd be curious to know your thoughts and ideas about how you'd go about solving this.
from cf-deployment-concourse-tasks.
I'd like to see a flag to disable committing to git, or ideally one to enable it but that would break users.
Right now the error doesn't fail our jobs so it is just a bit of noise. @wfernandes wrote a vault-resource that tarballs up the bbl state dir and crams it into vault. That is how we are managing it.
I fear the git error message encourages using the git-resource to store bbl state dir, which, like I said previously can be risky.
Also, I have heard rumors of bbl supporting storing state securely using something like credhub and not hitting disk. I'm not fully up to speed with that development however.
from cf-deployment-concourse-tasks.
We have created an issue in Pivotal Tracker to manage this:
https://www.pivotaltracker.com/story/show/159270439
The labels on this github issue will be updated when the story is started.
from cf-deployment-concourse-tasks.
The cloudfoundry/bbl-state-resource may suggest an alternate path forward, using storage buckets on GCP.
from cf-deployment-concourse-tasks.
We don't currently have plans to store state in credhub. Bosh CLI may one day support secret storage during create-env, which would extend to BBL for free, but I don't believe that is under active development.
We do plan to eventually support storage in all major IaaS vendor's blob storage (and already do support it for S3) which would remove the temptation to use Git for this. You still have to opt in, but when opted in your state is always synced with the bucket and won't be on disk (outside of tmp) unless you pull it down.
Wherever you store your state, the critical items for ephemeral/integration/non-PII environments to not store on disk would be the IaaS credentials, which already can be stored in credhub (or Vault) via a Concourse's built in secret management. I think it is redundant to add the configuration and bosh credentials to Vault if you are a CF component team, these should be rotating on a daily basis.
from cf-deployment-concourse-tasks.
Haven't tried it yet, but what about something like git-crypt/sops? It's still committing to git, but encrypted with gpg keys. Not sure how well it'll work, I'm going to try it out in the next few days. Doesn't really solve the root concern since it's still in git, but figure I'd share the idea.
from cf-deployment-concourse-tasks.
Hi @jasonkeene
We recently made a change to the bbl-up
and bbl-destroy
tasks to allow users to optionally store the bbl-state
as a tarball that can be persisted in S3/GCS/online storage instead of committed to a GH repo. If this is this a suitable solution, could you please close this issue so we know that we don't need to address it further in future?
Thanks,
Dave
from cf-deployment-concourse-tasks.
^if using s3 for storing state as a tarball, is it possible to consume it as a tarball as input
in other tasks?
from cf-deployment-concourse-tasks.
Hi @aegershman,
Yes. Both s3
and gcs
resources support an unpack: true
parameter that will unpack a tarball during the get
step.
Regards,
Dave
from cf-deployment-concourse-tasks.
@jasonkeene, Let us know if the last response from Dave solves the issue you described. I am closing due to inactivity but you can always reopen the issue if the problem still persists.
from cf-deployment-concourse-tasks.
Related Issues (20)
- `update-integration-configs` should fail fast when `admin_password` is not found HOT 5
- bosh-deploy-with-created-release task should default to `name` if `final_name` is unavailable HOT 5
- bbl 8.4.40 is unable to fetch terraform openstack provider HOT 2
- Require Linux-ARM64 release for cloudfoundry/cf-deployment-concourse-tasks HOT 3
- Why do we install and then remove software-properties-common in cf-deployment-concourse-tasks dockerfile? HOT 2
- Update or remove the `notify-bbl-updates` job HOT 2
- null bug in delete-deployment task HOT 2
- Bump to latest patch of golang in dockerfile HOT 2
- Bump to go1.21
- SYSTEM_DOMAIN sometimes incorrectly derived from "toolsmiths-env" input
- build-docker-image job runs twice every time
- Bump to go1.22
- The task run-cats failed with "Not logged in." HOT 2
- run-errand task should support the `when-changed` flag HOT 2
- Discussion: Merge `upload-stemcells` and `deploy` tasks HOT 3
- cf tail command not found in docker image. HOT 5
- Delete Deployment / Destroy Director flag for no fail HOT 3
- release notes incorrect for v7.15 HOT 2
- Pushing creds to credhub rather than git HOT 3
- collect-ops-files task should create $BASE_OPS_FILES_DIR HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cf-deployment-concourse-tasks.