genesis-community / genesis Goto Github PK

We should provide a stub settings.yml file to genesis ci repipe, and instruct users to fill out params there, rather than failing gracefully on a spruce merge when that file doesn't exist.

Right now, the --template behavior prints nothing. This differs from the non-templated behavior significantly (where you get to see all of the files created).

We should tree the directory, and point out that we created the environment from a template.

Instead of getting a useful message like This genesis deployment requires the '${cmd}' command, you get

which: no safe in (/home/centos/.rvm/gems/ruby-2.1.8/bin:/home/centos/.rvm/gems/ruby-2.1.8@global/bin:/home/centos/.rvm/rubies/ruby-2.1.8/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/centos/perl5/bin:/usr/local/go/bin:/home/centos/go/bin:/home/centos/.local/bin:/home/centos/bin:/home/centos/.rvm/bin)

Tried genesis new deployment -t bosh-init --template bosh my-deployment-name today, and instead of making it a bosh-init deployment, it made it a regular deployment, though it did pull the templates properly. The -t flag should still set the deployment type on the new deployment repo. If any type was provided via upstream templates, -t should override that.

If there is a bin/genesis file in the repository, the Makefile should use that, instead of requiring the user to install one.

If not present, or if USE_SYSTEM_GENESIS is set in the environment, use whatever we can find in $PATH

Right now, there isn't any formal documentation about when Genesis Index is contacted, or how things like track, latest, and 1.3.4 versions should behave. This makes it hard for people to anticipate what genesis will do under certain circumstances. We should add some, to set expectations.

Genesis "help" is missing some of the cammands needed for normal operations like how to add a release

$ genesis help
USAGE: genesis <command> [arguments]

Common commands:

    genesis new deployment      Create a new deployment directory, with all the
                                necessary files for deploying with Genesis.

    genesis new site            Set up the files and directories for a new site.

    genesis new environment     Set up the files and directories for a new environment.

    genesis build               When run from inside of an environment directory,
                                paste together all of the template files, in order
                                and pop out a manifest file.

    genesis diff                When run from inside of an environment directory,
                                show a semantic diff of the generated manifest for that
                                environment, against some other manifest file.

    genesis refresh             Refresh the cached copies of site and/or global
                                definitions inside the current environment directory.
                                Can also be used to recreate the Makefile after a
                                Genesis upgrade.

    genesis bosh                Generates a deployable manifest (which may have secrets
                                in it) and then runs the `bosh` CLI with remaining
                                arguments.  Useful for recreating workers.

    genesis help                Peruse the help!  Give it a topic argument, or the name
                                of another command to get more detailed information.
                                Documentation!

$ genesis help add
Unrecognized help topic 'add'.
Try one of these:

    genesis help usage
    genesis help <command>

$ genesis add release rabbitmq
Using the latest version of rabbitmq available via the Genesis Index
$

We should make sure known commands show up in at least the initial help page

We should cache whether the director was reachable in the director_alive function, so it only has to check once per genesis invocation. Having make deploy check the director each time it tries to get release/stemcell info is leading to successes for the first few releases, and then failure for no good reason other than a VPN hiccup.

the global/release/<release>/sha1 is not being updated correctly when release version track is being used.

EG: https://ci.vsphere.starkandwayne.com/pipelines/dingo-postgresql-deployments/jobs/bosh-lite-try-anything/builds/83

When generating new normal sites, we should run bosh status --uuid and embed that in director.yml

We should also take the name of the site, environment and deployment and put those in name.yml thusly:

name: $site-$env-$deployment

That way people have less to configure when they create new environments.

There are some paths in vault that we would like to specify in site/properties.yml that differ only by the 'environment' section of the path.

eg: (( vault "secret/aws/" meta.environment "shield/something:secret" ))

Could the environment name be added to the name.yml file by default on environment creation? I think this could be generally useful.

These are things I have run into and changed locally while trying to get the genesis pipeline working. This issue is a reminder to create a PR with the changes

• CI_VAULT_APP_ID is not being set (VAULT_APP_ID is)
• use curl -k when authenticating to vault from pipeline
• export VAULT_TOKEN before vault secret/handshake

The following command prompted me for github username/password.

genesis new deployment --template cf_deployment cf

The description below for the template parameter does not make it clear that the _deployment cannot be specified on the command line. Either improve the documentation or allow _deployment to be written in the template parameter.

The command that worked was

genesis new deployment --template cf

The NAME parameter looks to be required in the help but actually it was optional if the --template parameter is specified.

USAGE: genesis new deployment [options] NAME

  Create a new deployment, complete with all the correct templates in
  global/, and a base configuration.  This will create a sub-directory
  in the current working directory, named <NAME>-deployments, and set
  it up with git version control.

  OPTIONS

    -t, --template <name>

        Base the deployment off of an upstream Genesis template, by
        cloning a repository from github.com and using that.

        Templates can be specified as qualified name containing the
        Github org/user and repository name, i.e. "jhunt/bolo", or
        as a simple name, i.e. "shield", which will be inferred to
        belong to the starkandwayne organization.

Unlikely to happen under normal scenarios, but nice to handle the edge case gracefully

Just like genesis add release does

genesis ci beta eu-west-1/proto uses a regex that fails because it's not an anchored - check.

Line 2676 shows [[ ${site_env} =~ -.* ]] && bad_usage ${USAGE} as an example.

The CI pipeline as generated currently does not attempt to run any sort of smoke tests against the BOSH deployments being automated. This makes it difficult to auto-propagate to production environments safely. Just because it deployed doesn't mean it works.

Introduce some way of specifying smoke tests to run on a per-global / per-site / per-environment manner.

Probably just a smoke-tests script in the env / site / global directory?

Currently it does nothing (exit 0)

• Implement it, using either spiff diff (workable) or spruce diff (requires work on Spruce)
• Write the help topic genesis help diff
• Add it to the Makefile for environments

It would be nice if genesis use stemcell would take aliases, like 'vsphere' or 'aws', and pick a suitable default.

genesis use stemcell vsphere 3177
genesis use stemcell warden 2776

etc.

I propose curating a starkandwayne/genesis image for the pipelines genesis creates. I don't see the benefits of reusing starkandwayne/concourse.
AFAIK starkandwayne/concourse is being used across multiple projects of different kinds and is changing for reasons independent of the needs of genesis. There may be some merit in de-duplicating images across similar looking projects but since genesis is a very focused tool I would prefer having all resources it depends on as isolated / explicit as possible.

Not sure if genesis needs fixing, or the help needs fixing.

Got sidelined due to some weird errors regarding static IPs already allocated, but we couldn't find the issue when we traced the yml.

Turns out that this is a problem with using an older copy of spruce. Genesis should verify that the version of spruce is up to date, not that it simply exists

This feature will show what the global releases are (including version). If executed in site scope, will show if those releases are used by that site. If executed in env scope, will show differences in versions if any.

After a fresh make site and env to parallel a existing site, we copied the stem cells folder over (to fix Error: no stemcell name specified for site) but then we get this error when we do a test make

[ruby-2.2.3] (master) Homer:sandbox krutten$ make manifest
  checking https://genesis.starkandwayne.com for details on stemcell bosh-azure-hyperv-ubuntu-trusty-go_agent/3262.2
Error: release 'README.md' not defined globally
make: *** [manifest] Error 2
[ruby-2.2.3] (master) Homer:sandbox krutten$

bosh micro is no longer a thing, so we should just remove the code altogether.

doing google searches to figure out if a template exists seems slightly awkward. Knowing aws/vsphere, deployment packages and maybe releases would be nice to know in the template readme file.

I think this line is preventing the command from working as expected https://github.com/starkandwayne/genesis/blob/master/bin/genesis#L2426

Hi! The repository doesn't currently contain a license...wondering if that's intentional? Thanks!

(Thanks to @sushiandbeer for catching this!)

We seem to trigger building a new version of the genesis docker image every time we ship genesis, but the genesis script included in the image is hard-coded to 1.5.2. Should this be updated to pull in the latest GH release genesis, and make use of that, so we can auto-trigger docker builds based off new genesis versions?

ci/pipeline.yml will contain the pws from boshes.yml in plain text.

Allow genesis to always update to the latest version of an indexed release/stemcell by specifying the 'track' keyword. 'latest' is currently used by bosh to specify the latest version of that release/stemcell on the bosh director, so we don't want to overload that keyword in genesis. If genesis determines the version to be track, in an indexed release/stemcell, it should contact the index at manifest generation time to grab the latest version/url/sha1. If it's track in a non-indexed genesis deployment, it should convert the version to latest, so that bosh will know what to do

Build a small CF app for tracking versions / URLs and SHA1s of BOSH releases.

There should be more/better help and documentation about genesis ci, what it does, how the pipelines are built/configured/uploaded, what types of pipeline nodes (alpha,beta, etc) exist, and how they all interact

Currently genesis contacts the index during cmd_build(). This has 2 negative effects:

• a lot of commands run cmd_build twice in a row (once redacted once un-redacted) and it is annoying to wait that long twice during the same command flow.
• in a pipeline things can get out of sync. eg when using track it is possible for a beta environment to deploy a later version of a release than the alpha environment because the version is resolved while building the manifest.

Suggestion:
Only resolve releases when deploying in alpha environment.
Only resolve stemcells in alpha and beta environment.
All other environments copy the data and don't attempt to resolve to index

We should retrieve/cache stemcell information from the director in one-shot, and then parse the cached copy out via jq to determine if releases/stemcells are present.

I think the easiest way to do this is to iterate over a copy of the $@ arguments at boot, and if we see a help flag, to recast the command as genesis help $@

All sub-commands should do some basic argument processing, preferably all the same way. They should catch -* as an invalid sequence, but also handle -- properly. If someone really wants a --thing deployment repo, or --minus site, I'm game.

If you are in a site directory (or below it) genesis new env NAME should Do The Right Thing ™️

Please add the ability to upgrade genesis in place, via command. Ideally something like genesis update would wget down the latest release, move the old version to .bak and drop in the latest version.

Huge kudos if we can get genesis to also update spruce and safe to the latest builds.

Thanks!

It would be nice if genesis could provide a stub ci/boshes.yml to be filled out, rathe than complaining that it needs to be filled in, and not providing any assistance.

Genesis does not check for availability of tree, but without it, genesis new (site|env) commands fail during final output.

Would like to show the difference between what is currently in .global and .site from the global and /site dirs that would be applied when make refresh is run. Currently this is doable by running make refresh, doing the git diff, then git checkout .global .site to backout those changes you don't want.

resource_pools.network is a required field - the network name should be cf

Doc:
https://bosh.io/docs/deployment-manifest.html#resource-pools

When the the genesis pipeline's build-docker-image step fails, it causes the shipit job to fail, even if a GH release was created. This makes it difficult to re-run the shipit job, as it has already been shipped, and release notes have been cleaned up. Can the docker image be separated out into another job?

After #24, we should be able to remove the differences between normal bosh and bosh-init template files, and not have to merge/gen differently

I have not had a chance to track down which genesis release breaks this, but as of the latest update, running "make deploy" generates the following error:

$ make deploy
Error: Could not contact https://<director IP>:25555 to find release information
make: *** [deploy] Error 3

This appears to be coming from this code block

have_blob() {
    local type=$1; shift
    local name=$1; shift
    local version=$1; shift

    local director_url=$(spruce json ~/.bosh_config | jq -r ".target")
    local director_creds=$(spruce json ~/.bosh_config | jq -r '.auth[.target].username + ":" + .auth[.target].password')
    if director_alive $director_url; then
        if [[ ${version} == "latest" ]]; then
            if [[ ${type} == "release" ]]; then
                if [[ -n $(curl -Lks -u "${director_creds}" ${director_url}/releases | jq -r '.[]| select(.name == "'${name}'") | "release-detected"') ]]; then
                    return 0
                else
                    return 1
                fi
            elif [[ ${type} == "stemcell" ]]; then
                if [[ -n $(curl -Lks -u "${director_creds}" ${director_url}/stemcells | jq -r '.[]| select(.name == "'${name}'") | "stemcell-detected"') ]]; then
                    return 0
                else
                    return 1
                fi
            else
                echo >&2 "Error: Invalid type. This is a bug. Got ${type} but expected 'stemcell' or 'release'"
                exit 2
            fi
        else
            if [[ ${type} == "release" ]]; then
                if [[ -n $(curl -Lks -u "${director_creds}" ${director_url}/releases | jq -r '.[] | select(.name == "'${name}'") | .release_versions[] | select(.version == "'${version}'") | "release-version-detected"') ]]; then
                    return 0
                else
                    return 1
                fi
            elif [[ ${type} == "stemcell" ]]; then
                if [[ -n $(curl -Lks -u "${director_creds}" ${director_url}/stemcells | jq -r '.[]| select(.name == "'${name}'") | select(.version == "'${version}'") | "stemcell-version-detected"') ]]; then
                    return 0
                else
                    return 1
                fi
            else
                echo >&2 "Error: Invalid type. This is a bug. Got ${type} but expected 'stemcell' or 'release'"
                exit 2
            fi
        fi
    else
        echo >&2 "Error: Could not contact ${director_url} to find release information"
        exit 2
    fi
}

Interestingly enough, "genesis bosh deploy" still works (even though our Makefile uses genesis).

Allow specification of per-environment deployment types (bosh vs. bosh-init).

genesis new env will need a new flag, --type, for setting that.

Update the dependency hook checking to differentiate between binaries that are always needed, and binaries that are needed just when running env hooks, to make life easier for concourse deployments + task docker images

Currently when site/stemcell/* changes the deployment will fail and it is necessary to manually upload the bosh stemcell.