Our travis builds tolerate more warnings than is ideal. For example warnings of the form

pkg/git/mock/mock.go:203:3: Fatalf call has error-wrapping directive %w

seen when running go test ./... locally are tolerated silently. Add golangci-lint to our Travis build. See the script section of https://gist.github.com/y0ssar1an/df2dab474520c4086926f672c52db139 for an example.

Each promotion PR needs its own branch. The CLI user should not be required to specify --branch [new branch] on each request. If --branch is not specified, a temporary, unique branch should be created.

As well as building a docker image, we would like to make the services binary available for download. Ideally we'd like binaries for Linux, Mac and Windows. Can we get Travis to build and publish them back to GitHub as downloads for a release?

(i.e. push a tag to GitHub, create a release, Travis release plugin publishes artifacts.)

The bootstrap branch in https://github.com/rhd-gitops-example/odo shows how odo pipelines bootstrap (init) can be used to create a gitops repo that should be compatible with our promote verb. Check that this is still the case. Can we script a manual test for this?

We now have docker images published to quay.io/redhat-developer/gitops-cli. Re-test the Tekton sample in docs/ with this image, and update the yaml and readme once it's confirmed to be working.

I've had a look through the code and readme and noticed https://github.com/rhd-gitops-example/services/blob/master/pkg/avancement/service_manager.go#L64

So I assume deployment.txt is just an example of what we're using atm as a PoC, where do we think this should do instead going forward?

I'm thinking it should pickup whatever folks we'd like, perhaps specified via some args and/or a file, so discussion welcome

(nit: should the package be advancement not avancement?)

The reproduce for this behaviour is to follow https://github.com/rhd-gitops-example/services/tree/master/tekton-example and do the first PipelineRun creation, then the second TaskRun creation.

You'll notice that for the service-promote Task, we get a pull request mentioning a local filesystem.

I believe this is because we clone the project to the workspace Tekton uses so it's not a valid URL (it's, for example, /workspace/git-app-repo and not https://github.com/myorg/repo.git), and thus we assume the user is going down a local promotion path.

This isn't what I expected though, I thought it would be a promote from repository X to repository Y.

The problem PR: https://github.com/a-roberts/gitops-test/pull/2
And not a problem PR (from repo to repo), you get a-roberts/gitops-repo-testing#1 which is better.

Note, I've opened an issue to use exemplar repositories (both in terms of name and contents) for the Tekton example at #68.

https://github.com/rhd-gitops-example/services/blob/master/tekton-example/resources/build-task.yaml mounts the docker socket. Replace this with buildah as per https://github.com/tektoncd/catalog/tree/master/buildah

odo runs on Windows and we're close I think to having Windows binaries/tests being done, see https://travis-ci.org/github/rhd-gitops-example/services/jobs/678499631 which indicates classic portability problems (differences in path separators and newlines)

Under this issue - using Travis as the dev/test environment if you must, add windows into the .travis.yml as an operating system and fix up our tests. Bonus points if you do manual testing as well to check our test coverage is good and all is well 😄

Copying a comment from @mnuttall on Slack when I asked about this (so, incase it's not easy):

One IBM colleague (Tom Seelbach) noticed that we weren’t using a viper/cobra base. He mentioned that they had created their CLI project by copying odo and reducing it to a shell , and that that framework gave them multiplatform support very easily. I mention it in case Windows-compatibility becomes difficult on the current code base.

In testing the https://github.com/rhd-gitops-example/services/tree/master/automerge-example @dibbles noticed errors of the form

HEAD detached from d88df61
[do-this-first : merge-pr] nothing to commit, working tree clean

when using Tekton Pipelines 0.12. These errors are not seen with Pipeline 0.11.3. The PipelineResources in the automerge-example need to be updated and re-tested with Tekton 0.12+. The failure is believed to be due to an optional refspec option introduced in the 0.12 timeframe. See tektoncd/pipeline@8bc424d

revision: Git revision (branch, tag, commit SHA or ref) to clone. You can use this to control what commit or branch is used. git checkout is used to switch to the revision, and will result in a detached HEAD in most cases. Use refspec along with revision if you want to checkout a particular branch without a detached HEAD. If no revision is specified, the resource will default to master.

Fetch the develop branch and switch to it (not detached)
  revision: develop
  refspec: refs/heads/develop:refs/heads/develop 

Ensure that both the 'standalone' and 'webhooks' PipelineResources are updated and tested.

• Split a DEVELOPMENT.md out of the readme.
• Introduce the tool, link to documention of the repository structure - currently that's available under https://github.com/rhd-gitops-example/manifest-documentation#configuring-services
• Document where to get the cli binary from, and what its options do
• Be careful to differentiate the options in --from between local and remote cases.

We need to embed the promote cli in a Tekton task. A typical pipeline should be,

• docker build
• docker tag
• docker push
• promote [newly built image] [into dev]

Build a Tekton Task and Pipeline that does this, and demonstrate it back to the team.

#32 looked good but it broke promotion from a local repository. We lack the tests for the main Promote() method that should have caught it.

Promote() results in the generation of a PullRequest. A test for this method might need to generate pull requests for both local and remote scenarios, check that they look right, and then close the PRs.

An alternative would be to introduce a --dry-run CLI flag so that we could verify what the PR would look like, were we to raise it, but stop short of doing so. This might make the tests a bit more reliable.

Accept URLs of the form, https://[email protected]/username/repository.git and https://username:[email protected]/username/repository.git. Don't prompt for username or password if they have all ready been provided.

More often than not, 'password' should be a token, and 'username' won't matter.

Let's not surface this:

(base) Adams-MBP:services aroberts$ ./services promote --from https://github.com/a-roberts/gitops-repo-testing --to https://github.com/a-roberts/staging --service service-a
2020/03/31 11:12:44 failed to clone source repo https://github.com/a-roberts/gitops-repo-testing: failed to clone repo https://promotion:(token here)@github.com/a-roberts/gitops-repo-testing: exit status 128

Instead let's provide a custom error message for this scenario - not cool when, say, I'd want to demo something, accidentally do the wrong command or miss a pre-req, and now my token's there for the world to see!

Expected Behaviour

When I run a promote command with --debug I should see additional output at the point of failure of the command

Actual Behaviour

(base) Megans-MacBook-Pro:services [email protected]$ ./services promote --from https://github.com/Megan-Wright/gitops-repo-testing.git --to https://github.com/Megan-Wright/staging.git --service service-a --debug --branch-name megan-test-6****@@@@@
2020/04/08 13:22:39 Cloning into 'gitops-repo-testing'...

2020/04/08 13:22:41 Cloning into 'staging'...

2020/04/08 13:22:41 failed to checkout repo: failed to checkout branch megan-test-6****@@@@@ in repo https://github.com/Megan-Wright/staging.git: exit status 128

(which is the same output you get from running the command without --debug)

Steps to Reproduce

Run a promote command with --debug which will obviously fail, such as adding invalid characters to the branch name like I have done above

@bigkevmcd writes,

oc has been released with plugin support
so, if you were to rename the cmd/services to cmd/oc-services or symlink it or do something else you'd have a plugin

Test this out! Does oc add any value to running services either in a docker container on the command line?

With 379c24b merged it'd be useful to have the PipelineRun/TaskRun in the example being runnable through tkn as opposed to creating the PipelineRun from yaml.

We believe this is blocked by tektoncd/cli#940 and this is the command we have so far that'd we like to prove works and then document.

tkn pipeline start service-promote-pipeline --resource git-source=git-app-repo --resource docker-image=docker-app-image --param commitId=v1 --param github-secret=promote-secret --param github-config=promoteconfigmap --param to="https://github.com/Megan-Wright/staging.git" --param service=service-a --workspace name=repo-space,secret=promote-secret

Builds of each integration should be published somewhere official? Might be something for @bigkevmcd and @mnuttall to discuss depending on the future of this code

• will it be integrated with odo or standalone?
• does it need a short term home whilst things are decided?

Expected Behaviour

When I run a promote command which fails, if I then amend this to a promote command which should pass and try to rerun it, I expect it to work without error.

Actual Behaviour

After a failed promote command, any new promote command (whether valid or not) will fail because the cache hasn't been cleared, and so I have to run rm -rf ~/.promotion/cache; before subsequent promote commands.

Steps to Reproduce

Run a promote command which will fail, such as adding invalid characters to the branch name, and then immediately rerun the same promote command in a way which should work.

The sample in the docs directory has a hardcoded name for the pipelinerun meaning you cannot reapply the yaml and generate new runs. Can this not be changed to generateName rather than name in the metadata?

It should now be possible that on pushing a tag starting with v, a release is made with binaries uploaded as assets, the job currently relies on GH_TOKEN being set.

odo pipeline init creates GitOps repos that use a modified version of the gitops repository spec, as per https://github.com/rhd-gitops-example/manifest-documentation. See for example https://github.com/bigkevmcd/gitops/.

In order to support multiple environments, a path /environments/environment-name/ is added to the top of the directory structure.

Currently we use

services promote --from [path] --to [git url] --service [name] 

This means, copy [name]/config/* to /services/[name]/base/config/* in the [git url] repo via a PR. In the new case, the target path is /environment/[environment-name]/services/[service-name]/base/config/* and the question is, How do we determine [environment-name] when preparing the pull request, if the [service-name] subdirectory does not exist?

• We could add another command line arg, --env env-name
• We could say, ‘if /environment contains only one subdirectory, use it’
• We could say, ‘if /environment is empty, create a subdirectory whose name is the same as the target repo’

Under this issue implement option 2 and either implement option 1 or raise a follow-on issue to do so.

Also update our example projects (gitops-exampl-dev and -staging) to conform to the new format.

Also update our sample code and documentation.

Further to #75 investigate why GitHub Enterprise Pull Requests don't go into 'merged' state when a merge commit is pushed, leaving code in automerge-task.yaml of the form,

# On GitHub Enterprise - not GitHub.com - the issue will still be open. 
# We don't know if this is a bug or a feature. One way to work around this is with code of the form, 
# export GITHUB_HOST=github.ibm.com
# hub api -XPATCH repos/mnuttall/gitops-example-dev/pulls/4 -f state=closed
# See https://github.com/github/hub/issues/1151

If unavoidable, replace the commented out section above with working code that parses the PullRequest URL to generate the hub api -XPATCH command correctly.

Most of our testing so far has been on public GitHub. Re-test against GitHub Enterprise.

Replace --branch-name with two options:

--from-branch    # branch in source repo to promote from
--to-branch.     # branch in destination repo to promote to 

If both options are specified and --to is not set, set --to to be equal to --from.

The initial cache implementation fetches the default branch for a repository - this may not be --from-branch, so that may also need fixing under this issue.

We're using the 'github' go-scm driver to create PullRequests. Some additional work is required to add GitLab support.

I'll do this one, going with a simple Go app instead. Things take a minute with the simple Pipeline atm and 16 minutes with the Buildah one I've got in a PR.

With the example I'll propose, it's one minute for the Buildah Pipeline.

Review the instructions in the doc directory and simplify the instructions where/if possible - additional comments might be added to yaml files or classifications made to make it easier for a novice to know what needs changing.

Discuss: consider extending cli to permit promoting more than one service in a single PR. (e.g. new service-a depends on new service-b.)

The sample in the docs directory contains a number of yaml files which are not required. Remove these files.

When following the example using Tekton 0.11, all was going well until

(base) Adams-MBP:docs aroberts$ k apply -f servicepromotepipelinerun.yaml 
error: error when retrieving current configuration of:
Resource: "tekton.dev/v1beta1, Resource=pipelineruns", GroupVersionKind: "tekton.dev/v1beta1, Kind=PipelineRun"
Name: "", Namespace: "promote"
Object: &{map["apiVersion":"tekton.dev/v1beta1" "kind":"PipelineRun" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":""] "generateName":"service-promote-pipelinerun-" "namespace":"promote"] "spec":map["params":[map["name":"commitId" "value":"v1"] map["name":"github-secret" "value":"promotesecret"] map["name":"github-config" "value":"promoteconfigmap"] map["name":"to" "value":"https://github.com/<your github org>/<your github repo>.git"] map["name":"service" "value":"service-a"]] "pipelineRef":map["name":"service-promote-pipeline"] "resources":[map["name":"git-source" "resourceRef":map["name":"git-app-repo"]] map["name":"docker-image" "resourceRef":map["name":"docker-app-image"]]] "serviceAccountName":"demo" "workspaces":[map["name":"repo-space" "persistentVolumeClaim":map["claimName":"repopvc"]]]]]}
from server for: "servicepromotepipelinerun.yaml": resource name may not be empty

This is using Tekton 0.11 as proven with

        image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/controller:v0.11.0-rc2@sha256:97c8424c6091ab3ffe5a56c4c564e0342279f8c16f3b51ddc66bf3f22e457a78

As an extra bonus, don't use the now deprecated PipelineResources, see https://github.com/tektoncd/pipeline/blob/master/docs/migrating-v1alpha1-to-v1beta1.md#pipelineresources-and-catalog-tasks which mentions the equivalent types we could use in Tasks.

Note that we may want to have a separate folder for the v1beta resources so anyone stuck on alpha for whatever reason could still use that - something to think about.

A sample gitops directory structure can be found in https://github.com/mnuttall/gitops-repo-testing. (This should itself be cleaned up and moved into the rhd-gitops-example org.)

Commands such as

services promote \
 --from https://github.com/mnuttall/gitops-repo-testing \
 --to   https://github.com/mnuttall/staging \
 --service services/service-a

creates a PR like https://github.com/mnuttall/staging/pull/1/files in which the copied files in under a root directory, 'service-a'.

What we want instead is for promote --service service-a to effectively cp -R services/service-a from the source to the target repository.

This enforces a convention that services are in a directory called 'services'. That's ok.

See also https://github.com/rhd-gitops-example/services#running - these docs need fixing.

The CopyService() function in https://github.com/rhd-gitops-example/services/blob/master/pkg/git/copy.go , called from

We currently publish binaries in our release process, but we also need to publish docker images.

Work with RedHat to establish whether we can publish specifically tagged images to quay.io from our travis.yml, or whether a different mechanism is required.

kubectl apply -k git-source/env is currently used, so fix up that

In the first instance we need to go build and go test on PullRequests and merges. Travis is the simplest way to get this up and working.

go build, go test are broken further to the move of git repo.

cd $GOPATH/src/github.com/rhd-gitops-example/services
go build ./cmd/services
cmd/services/main.go:9:2: cannot find package "github.com/bigkevmcd/services/pkg/avancement" in any of:
	/usr/local/Cellar/go/1.12.6/libexec/src/github.com/bigkevmcd/services/pkg/avancement (from $GOROOT)
	/Users/[email protected]/dev/goprojects/src/github.com/bigkevmcd/services/pkg/avancement (from $GOPATH)
cmd/services/main.go:10:2: cannot find package "github.com/bigkevmcd/services/pkg/git" in any of:
	/usr/local/Cellar/go/1.12.6/libexec/src/github.com/bigkevmcd/services/pkg/git (from $GOROOT)
	/Users/[email protected]/dev/goprojects/src/github.com/bigkevmcd/services/pkg/git (from $GOPATH)
cmd/services/main.go:11:2: cannot find package "github.com/mitchellh/go-homedir" in any of:
	/usr/local/Cellar/go/1.12.6/libexec/src/github.com/mitchellh/go-homedir (from $GOROOT)
	/Users/[email protected]/dev/goprojects/src/github.com/mitchellh/go-homedir (from $GOPATH)
cmd/services/main.go:12:2: cannot find package "github.com/tcnksm/go-gitconfig" in any of:
	/usr/local/Cellar/go/1.12.6/libexec/src/github.com/tcnksm/go-gitconfig (from $GOROOT)
	/Users/[email protected]/dev/goprojects/src/github.com/tcnksm/go-gitconfig (from $GOPATH)
cmd/services/main.go:13:2: cannot find package "github.com/urfave/cli/v2" in any of:
	/usr/local/Cellar/go/1.12.6/libexec/src/github.com/urfave/cli/v2 (from $GOROOT)
	/Users/[email protected]/dev/goprojects/src/github.com/urfave/cli/v2 (from $GOPATH)

 services promote --from promote-demo --to https://github.com/gitops-example-dev --service promote-demo

Note the lack of an org/owner.

This gives an error of the form

(base) Adams-MBP:gitops aroberts$ services promote --from promote-demo --to https://github.com/gitops-example-dev --service promote-demo
Error: failed to clone destination repository, error: failed to clone repository, error is: could not identify repository name: https://promotion:<my token here>@github.com/gitops-example-dev. repository URL = https://github.com/gitops-example-dev. repository URL = https://github.com/gitops-example-dev. repository URL = https://github.com/gitops-example-dev

This issue will affect #6, which may not be able to complete without it.

This issue relates to cases in which the config yaml for a given microservice lives with the source code. A (Tekton) pipeline that builds this source repo will,

• docker build
• docker tag
• docker push
• "promote config/* to the configured 'dev' gitops repo"

This latter step involves raising a pull request that takes config/* from the checked out version of the source repository and 'promotes' it to /services/service-name/base/config/* in the target gitops repository - frequently referred to as 'dev'.

In the first case we can assume that service-name is the name of the source repository.

Currently, promote --from /path/to/local/repo --to ur.to.target.repo --service svc-name will look for /services/svc-name/base/config/* in the source repository, since that is the behaviour for when promoting between repositories.

We need the CLI to differentiate between promoting between gitops repositories, and from a source repo into a gitops repo. This was discussed at length in Slack today. We may also later need to differentiate between uploading service config, and that of an application.

For a first pass under this issue, we need to decide between:

• Add new CLI arguments, or a new command, for this situation, or
• When --from is a local path, assume that we're in this mode, and so copy /path/to/local/repo/config/* to url.to.target.repo/services/svc-name/base/config/*.

Errors such as those in the git layer currently fail with messages such as

exit status 1

This is due to our reporting stderr but swallowing stdout in functions such as execGit(). Under this issue, add a --debug flag to services that prints out any stdout we are currently hiding, reports parsed command line arguments, etc. One of the standard Go loggers should be used for this purpose.

Promotion currently yields PRs with a commit message, 'This is a commit message.'

Improve this to make best use of the information available. We should be able to make use of the --service argument, and --from when it's a URL.

I'm going through it now and it's particularly clunky but can easily be improved - I'd rather set a bunch of variables once, have the variables be passed through to the yaml (under a templates folder perhaps), and be good to go - or simply move everything I need to apply to be under the one folder (instead of docs) and everything to change goes in another.

One fancy approach would be to have a bash script, a variables file, and then do some sedding instead of having users replace xxx in six files.

Tidy the documentation up and make it as short, clean, and, in one go, kubectl applyable as possible, perhaps using tkn

When I try to do a services promote command to promote to a gitops repo that's in an org, the command fails with the following error:

2020/05/15 13:49:55 failed to create a pull-request for branch workspace-git-source-local-dir-838ea, error: Not Found. repository URL = https://github.ibm.com/kabanero-org-test/test1-gitops

It creates the branch with the expected updates but does not create the PR. I gave my token pretty much all possible access.

After one call to promote, it is necessary to rm -rf ~/.promotion/cache before the next. Modify the CLI so that the cache is deleted by default. Add a --keep flag to prevent cache deletion.

With the improvements made at #65, we're still referencing contributor's repositories - worse still, the example repositories don't conform to our GitOps structure. So we end up Docker building them when surely that isn't what we want. Ideally, you'd just need

https://github.com/rhd-gitops-example/gitops-example-dev
https://github.com/rhd-gitops-example/gitops-example-staging
and a third repository, https://github.com/rhd-gitops-example/gitops-example-prod.

Under this issue, modify the example to use these pipelines successfully - building and pushing the code to Dockerhub for example).

We currently reference

• https://github.com/a-roberts/gitops-repo-testing
• https://github.com/akihikokuroda/promote-demo
• https://github.com/akihikokuroda/gitops-test

While these are fine to be forked for testing, let's have something more official.

Further to #72 our sample gitops repositories https://github.com/a-roberts/gitops-example-dev and https://github.com/a-roberts/gitops-example-staging need to be updated with top-level paths /environments/dev and environments/staging respectively.

(base) Adams-MBP:services aroberts$ ./services --debug promote --from https://github.com/a-roberts/gitops-repo-testing --to https://github.com/a-roberts/staging --service service-a 
2020/03/31 11:27:59 fatal: destination path 'gitops-repo-testing' already exists and is not an empty directory.

(base) Adams-MBP:services aroberts$ ./services promote --from https://github.com/a-roberts/gitops-repo-testing --to https://github.com/a-roberts/staging --service service-a --debug
Incorrect Usage: flag provided but not defined: -debug

NAME:
   services promote - promote from one environment to another

USAGE:
   services promote [command options] [arguments...]

OPTIONS:
   --from value          source Git repository
   --to value            destination Git repository
   --service value       service name to promote
   --branch-name value   the name to use for the newly created branch (default: "test-branch")
   --cache-dir value     where to cache Git checkouts (default: "~/.promotion/cache")
   --commit-name value   the name to use for commits when creating branches [$COMMIT_NAME]
   --commit-email value  the email to use for commits when creating branches [$COMMIT_EMAIL]
   --help, -h            show help (default: false)
   
2020/03/31 11:31:28 flag provided but not defined: -debug

You can see I had to provide it at globally (so at the top level, ./services --debug) and this means I can't stick it on the end. There's been good discussion in Slack when I raised this (so @bigkevmcd and @mnuttall, lemme know if I'm misquoting you or I've missed something important!).

Kevin suggested we could move to using cobra (I think that's an aside), and even provided the change we'd want to make (so at the time of this issue - simply from the global to promote flag list):
https://github.com/rhd-gitops-example/services/blob/master/cmd/services/main.go#L37 to https://github.com/rhd-gitops-example/services/blob/master/cmd/services/main.go#L85

so I think this makes for a useful first issue for somebody if we did want to make it so.

As a developer (or Task in a Pipeline, etc) I need to be able to say, promote service-name from/path/to/local/git/repo --to https://remote.git.repo

Either --from needs to take a directory path as well as https:// urls, or we need a different CLI argument. I'd prefer the former but could accept the latter.

The automerge-example should provide working, minimal RBAC and an attached service account for both the 'standalone' and 'webhooks' cases. Update the 'Prerequisites' section of the README accordingly.

I'm looking to contribute to this area, so I built the code here and I've ran through the example project I've found - there's no connection just yet though.

So, we've got an exemplar project at https://github.com/mnuttall/gitops-repo-testing/tree/dev (which should be under this org I guess?) and it'd be great to have the readme updated here to work with such a repository so things are more concrete than copying the sole text file.

See #2 - it's super related and also mentions including an exemplar project here