genesis-community / genesis Goto Github PK
View Code? Open in Web Editor NEWA BOSH Deployment Paradigm
License: MIT License
A BOSH Deployment Paradigm
License: MIT License
I think the easiest way to do this is to iterate over a copy of the $@
arguments at boot, and if we see a help flag, to recast the command as genesis help $@
Just like genesis add release
does
ci/pipeline.yml will contain the pws from boshes.yml in plain text.
There should be more/better help and documentation about genesis ci
, what it does, how the pipelines are built/configured/uploaded, what types of pipeline nodes (alpha,beta, etc) exist, and how they all interact
It would be nice if genesis use stemcell
would take aliases, like 'vsphere' or 'aws', and pick a suitable default.
genesis use stemcell vsphere 3177
genesis use stemcell warden 2776
etc.
genesis ci beta eu-west-1/proto
uses a regex that fails because it's not an anchored -
check.
Line 2676 shows [[ ${site_env} =~ -.* ]] && bad_usage ${USAGE}
as an example.
the global/release/<release>/sha1
is not being updated correctly when release version track
is being used.
Allow specification of per-environment deployment types (bosh vs. bosh-init).
genesis new env
will need a new flag, --type
, for setting that.
It would be nice if genesis could provide a stub ci/boshes.yml
to be filled out, rathe than complaining that it needs to be filled in, and not providing any assistance.
Instead of getting a useful message like This genesis deployment requires the '${cmd}' command
, you get
which: no safe in (/home/centos/.rvm/gems/ruby-2.1.8/bin:/home/centos/.rvm/gems/ruby-2.1.8@global/bin:/home/centos/.rvm/rubies/ruby-2.1.8/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/centos/perl5/bin:/usr/local/go/bin:/home/centos/go/bin:/home/centos/.local/bin:/home/centos/bin:/home/centos/.rvm/bin)
The CI pipeline as generated currently does not attempt to run any sort of smoke tests against the BOSH deployments being automated. This makes it difficult to auto-propagate to production environments safely. Just because it deployed doesn't mean it works.
Introduce some way of specifying smoke tests to run on a per-global / per-site / per-environment manner.
Probably just a smoke-tests script in the env / site / global directory?
Currently when site/stemcell/*
changes the deployment will fail and it is necessary to manually upload the bosh stemcell.
Allow genesis to always update to the latest version of an indexed release/stemcell by specifying the 'track' keyword. 'latest' is currently used by bosh to specify the latest version of that release/stemcell on the bosh director, so we don't want to overload that keyword in genesis. If genesis determines the version to be track, in an indexed release/stemcell, it should contact the index at manifest generation time to grab the latest version/url/sha1. If it's track in a non-indexed genesis deployment, it should convert the version to latest, so that bosh will know what to do
Please add the ability to upgrade genesis in place, via command. Ideally something like genesis update would wget down the latest release, move the old version to .bak and drop in the latest version.
Huge kudos if we can get genesis to also update spruce and safe to the latest builds.
Thanks!
Hi! The repository doesn't currently contain a license...wondering if that's intentional? Thanks!
(Thanks to @sushiandbeer for catching this!)
When generating new normal sites, we should run bosh status --uuid
and embed that in director.yml
We should also take the name of the site, environment and deployment and put those in name.yml thusly:
name: $site-$env-$deployment
That way people have less to configure when they create new environments.
Would like to show the difference between what is currently in .global and .site from the global and /site dirs that would be applied when make refresh is run. Currently this is doable by running make refresh, doing the git diff, then git checkout .global .site to backout those changes you don't want.
This feature will show what the global releases are (including version). If executed in site scope, will show if those releases are used by that site. If executed in env scope, will show differences in versions if any.
Right now, the --template
behavior prints nothing. This differs from the non-templated behavior significantly (where you get to see all of the files created).
We should tree the directory, and point out that we created the environment from a template.
These are things I have run into and changed locally while trying to get the genesis pipeline working. This issue is a reminder to create a PR with the changes
CI_VAULT_APP_ID
is not being set (VAULT_APP_ID
is)curl -k
when authenticating to vault from pipelinevault secret/handshake
Not sure if genesis needs fixing, or the help needs fixing.
We seem to trigger building a new version of the genesis docker image every time we ship genesis, but the genesis script included in the image is hard-coded to 1.5.2. Should this be updated to pull in the latest GH release genesis, and make use of that, so we can auto-trigger docker builds based off new genesis versions?
There are some paths in vault that we would like to specify in site/properties.yml
that differ only by the 'environment' section of the path.
eg: (( vault "secret/aws/" meta.environment "shield/something:secret" ))
Could the environment name be added to the name.yml
file by default on environment creation? I think this could be generally useful.
We should provide a stub settings.yml file to genesis ci repipe
, and instruct users to fill out params there, rather than failing gracefully on a spruce merge when that file doesn't exist.
We should cache whether the director was reachable in the director_alive function, so it only has to check once per genesis
invocation. Having make deploy
check the director each time it tries to get release/stemcell info is leading to successes for the first few releases, and then failure for no good reason other than a VPN hiccup.
All sub-commands should do some basic argument processing, preferably all the same way. They should catch -*
as an invalid sequence, but also handle --
properly. If someone really wants a --thing
deployment repo, or --minus
site, I'm game.
Right now, there isn't any formal documentation about when Genesis Index is contacted, or how things like track
, latest
, and 1.3.4
versions should behave. This makes it hard for people to anticipate what genesis will do under certain circumstances. We should add some, to set expectations.
The following command prompted me for github username/password.
genesis new deployment --template cf_deployment cf
The description below for the template parameter does not make it clear that the _deployment cannot be specified on the command line. Either improve the documentation or allow _deployment to be written in the template parameter.
The command that worked was
genesis new deployment --template cf
The NAME
parameter looks to be required in the help but actually it was optional if the --template parameter is specified.
USAGE: genesis new deployment [options] NAME
Create a new deployment, complete with all the correct templates in
global/, and a base configuration. This will create a sub-directory
in the current working directory, named <NAME>-deployments, and set
it up with git version control.
OPTIONS
-t, --template <name>
Base the deployment off of an upstream Genesis template, by
cloning a repository from github.com and using that.
Templates can be specified as qualified name containing the
Github org/user and repository name, i.e. "jhunt/bolo", or
as a simple name, i.e. "shield", which will be inferred to
belong to the starkandwayne organization.
I think this line is preventing the command from working as expected https://github.com/starkandwayne/genesis/blob/master/bin/genesis#L2426
Build a small CF app for tracking versions / URLs and SHA1s of BOSH releases.
We should retrieve/cache stemcell information from the director in one-shot, and then parse the cached copy out via jq to determine if releases/stemcells are present.
After #24, we should be able to remove the differences between normal bosh and bosh-init template files, and not have to merge/gen differently
If there is a bin/genesis
file in the repository, the Makefile should use that, instead of requiring the user to install one.
If not present, or if USE_SYSTEM_GENESIS is set in the environment, use whatever we can find in $PATH
Currently it does nothing (exit 0
)
spiff diff
(workable) or spruce diff
(requires work on Spruce)genesis help diff
Got sidelined due to some weird errors regarding static IPs already allocated, but we couldn't find the issue when we traced the yml.
Turns out that this is a problem with using an older copy of spruce. Genesis should verify that the version of spruce is up to date, not that it simply exists
After a fresh make site and env to parallel a existing site, we copied the stem cells folder over (to fix Error: no stemcell name specified for site
) but then we get this error when we do a test make
[ruby-2.2.3] (master) Homer:sandbox krutten$ make manifest
checking https://genesis.starkandwayne.com for details on stemcell bosh-azure-hyperv-ubuntu-trusty-go_agent/3262.2
Error: release 'README.md' not defined globally
make: *** [manifest] Error 2
[ruby-2.2.3] (master) Homer:sandbox krutten$
resource_pools.network
is a required field - the network name should be cf
Doc:
https://bosh.io/docs/deployment-manifest.html#resource-pools
Genesis "help" is missing some of the cammands needed for normal operations like how to add a release
$ genesis help
USAGE: genesis <command> [arguments]
Common commands:
genesis new deployment Create a new deployment directory, with all the
necessary files for deploying with Genesis.
genesis new site Set up the files and directories for a new site.
genesis new environment Set up the files and directories for a new environment.
genesis build When run from inside of an environment directory,
paste together all of the template files, in order
and pop out a manifest file.
genesis diff When run from inside of an environment directory,
show a semantic diff of the generated manifest for that
environment, against some other manifest file.
genesis refresh Refresh the cached copies of site and/or global
definitions inside the current environment directory.
Can also be used to recreate the Makefile after a
Genesis upgrade.
genesis bosh Generates a deployable manifest (which may have secrets
in it) and then runs the `bosh` CLI with remaining
arguments. Useful for recreating workers.
genesis help Peruse the help! Give it a topic argument, or the name
of another command to get more detailed information.
Documentation!
$ genesis help add
Unrecognized help topic 'add'.
Try one of these:
genesis help usage
genesis help <command>
$ genesis add release rabbitmq
Using the latest version of rabbitmq available via the Genesis Index
$
We should make sure known commands show up in at least the initial help page
Tried genesis new deployment -t bosh-init --template bosh my-deployment-name
today, and instead of making it a bosh-init deployment, it made it a regular deployment, though it did pull the templates properly. The -t
flag should still set the deployment type on the new deployment repo. If any type was provided via upstream templates, -t
should override that.
Currently genesis contacts the index during cmd_build(). This has 2 negative effects:
track
it is possible for a beta environment to deploy a later version of a release than the alpha environment because the version is resolved while building the manifest.Suggestion:
Only resolve releases when deploying in alpha environment.
Only resolve stemcells in alpha and beta environment.
All other environments copy the data and don't attempt to resolve to index
If you are in a site directory (or below it) genesis new env NAME
should Do The Right Thing ™️
When the the genesis pipeline's build-docker-image step fails, it causes the shipit
job to fail, even if a GH release was created. This makes it difficult to re-run the shipit job, as it has already been shipped, and release notes have been cleaned up. Can the docker image be separated out into another job?
Update the dependency hook checking to differentiate between binaries that are always needed, and binaries that are needed just when running env hooks, to make life easier for concourse deployments + task docker images
Unlikely to happen under normal scenarios, but nice to handle the edge case gracefully
doing google searches to figure out if a template exists seems slightly awkward. Knowing aws/vsphere, deployment packages and maybe releases would be nice to know in the template readme file.
bosh micro
is no longer a thing, so we should just remove the code altogether.
I propose curating a starkandwayne/genesis image for the pipelines genesis creates. I don't see the benefits of reusing starkandwayne/concourse.
AFAIK starkandwayne/concourse is being used across multiple projects of different kinds and is changing for reasons independent of the needs of genesis. There may be some merit in de-duplicating images across similar looking projects but since genesis is a very focused tool I would prefer having all resources it depends on as isolated / explicit as possible.
I have not had a chance to track down which genesis release breaks this, but as of the latest update, running "make deploy" generates the following error:
$ make deploy
Error: Could not contact https://<director IP>:25555 to find release information
make: *** [deploy] Error 3
This appears to be coming from this code block
have_blob() {
local type=$1; shift
local name=$1; shift
local version=$1; shift
local director_url=$(spruce json ~/.bosh_config | jq -r ".target")
local director_creds=$(spruce json ~/.bosh_config | jq -r '.auth[.target].username + ":" + .auth[.target].password')
if director_alive $director_url; then
if [[ ${version} == "latest" ]]; then
if [[ ${type} == "release" ]]; then
if [[ -n $(curl -Lks -u "${director_creds}" ${director_url}/releases | jq -r '.[]| select(.name == "'${name}'") | "release-detected"') ]]; then
return 0
else
return 1
fi
elif [[ ${type} == "stemcell" ]]; then
if [[ -n $(curl -Lks -u "${director_creds}" ${director_url}/stemcells | jq -r '.[]| select(.name == "'${name}'") | "stemcell-detected"') ]]; then
return 0
else
return 1
fi
else
echo >&2 "Error: Invalid type. This is a bug. Got ${type} but expected 'stemcell' or 'release'"
exit 2
fi
else
if [[ ${type} == "release" ]]; then
if [[ -n $(curl -Lks -u "${director_creds}" ${director_url}/releases | jq -r '.[] | select(.name == "'${name}'") | .release_versions[] | select(.version == "'${version}'") | "release-version-detected"') ]]; then
return 0
else
return 1
fi
elif [[ ${type} == "stemcell" ]]; then
if [[ -n $(curl -Lks -u "${director_creds}" ${director_url}/stemcells | jq -r '.[]| select(.name == "'${name}'") | select(.version == "'${version}'") | "stemcell-version-detected"') ]]; then
return 0
else
return 1
fi
else
echo >&2 "Error: Invalid type. This is a bug. Got ${type} but expected 'stemcell' or 'release'"
exit 2
fi
fi
else
echo >&2 "Error: Could not contact ${director_url} to find release information"
exit 2
fi
}
Interestingly enough, "genesis bosh deploy" still works (even though our Makefile uses genesis).
Genesis does not check for availability of tree, but without it, genesis new (site|env) commands fail during final output.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.