monasca / pr-bot Goto Github PK

Currently new and removed modules aren't detected when softUpdateRepositories is run. This should, eh, happen.

The bot tries to auto-detect as much as possible, but this isn't always feasible. It should be possible to store per-repository and per-module configuration in git to specify various options:

• versioning policies
• dependencies to ignore
• notification settings or templates
• possibly extra user-defined plugins that should be run on repository events (e.g. run some js when a new issue is filed)

Something like http://mozilla.github.io/nunjucks/templating.html would be useful here.

Version tags (especially for Docker images) often contain additional metadata about that entry in the repository. In most cases (at least that we care about) these are usually dash-separated tokens in the version, like python:3.6-alpine3.6.

In general we shouldn't automatically propose updates that would switch between different tags or "release trains". As an example, the bot generated an update to monasca-helm: monasca/monasca-helm#185

In this case monasca-helm's client chart is using "released" image versions with standard semver tags (monasca/client:1.6.0 - we'd probably consider this a null tag) and the bot tried to update it to use monasca/client:master-20170918-185757 (master tag). Since monasca/client:1.7.0 was also published we should have used that release instead.

This introduces some problems:

• determining tags for a version depends on the version number formatting (see also, versioning policies from #15)
• we'd need to store multiple current versions for every module from every repository - latest version per tag
• we'd also (maybe later) need some way to specify how+when dependencies actually receive updates (related to per-module configuration from #15). Behaviors might include:
  • semver-patch: only update z in x.y.z
  • semver-minor: either y or z in x.y.z
  • semver-major: always use latest available semver
  • semver-* could have additional policy options:
    • tagPolicy: sticky (default) only updates to matching tags (e.g. no updates to master from null), none ignores tags (current behavior)
    • tag: just lock to a particular tag
  • datever: (or something) use the tag with the latest embedded timestamp (for the master-{date}-{time} case)
  • last-published: current behavior

We should have some ability to retry failed actions (say, HipChat notification timeouts) and - more generally - run tasks on a timer.

GCF doesn't have a good way to do this, unfortunately. We'll need to roll our own to some degree.

Support for plain Docker registries should be added. Right now dockerhub repositories determine current image versions using APIs only available from hub.docker.com.

At a minimum we need to support private docker registries (running https://hub.docker.com/_/registry/). This is (very) partially implemented now but needs to be finished. Current version detection will be somewhat worse.

There's also Docker Trusted Registry. It exposes a few more API functions and may be able to return timestamped tags. If the APIs match those on hub.docker.com we can add it to that plugin, otherwise it will need its own.

Consider supporting tracking and updating glide.yaml and glide.lock

With release plugins (#9) and repository hooks (#19) we have a decent story for managing the whole CI/CD process, at least on the backend. However, we still need to give users control over how and when a change is is built and released. Ideally this should integrate directly with GitHub issues/pull requests and we should avoid having any external UI.

One idea is for the bot to detect changed modules in a particular PR. It can then leave a comment on the PR users can interact with. Examples in followup comments.

(extracted from #15 (comment))

Since the PR bot is handling GitHub hooks anyway, it could be a convenient way to add repo-specific event handlers. In addition to repository and module specific configuration (#15), we could also let users drop in .js scripts:

Some thoughts:

• .bot/hooks/pull_request.js: runs on a PullRequestEvent
  • handle project-specific labels, e.g. a push label in monasca-docker
• .bot/hooks/status.js: runs on a StatusEvent
  • post a comment on the relevant PR with CI results, like a link to logs or a summary of any error messages
  • automatically merge on CI completion if a label is applied

We may want to execute these hooks in a sandboxed context, like vm2.

Flat git repositories should be supported (as per

'flat' in this case would mean a single module in a repository, presumably not in a subdirectory. Most single-project repositories (not like monasca-docker) are built like this, including the pr-bot repo itself.

Right now our HipChat notifications are very noisy. As an example, we just merged monasca/monasca-docker#322. This caused:

• 2 notifications on the initial commit for both travis events passing (push, pr)
• 2 notifications for another commit (push, pr)
• 1 notification on merge (push)
• 2 notifications for commit to monasca-helm (push, pr)
• 1 notification for merge to monasca-helm
• 1 notification for GitHub pages status update
• 1 notification for auto-update PR to monasca-helm/monasca
• 1 notification (pr) for Travis status update to monasca-helm for auto-update PR (no push since the bot commits to a fork)

So 1 change through the pipeline generated 11 notification cards in our HipChat room.

Issues with this include:

• the bot didn't auto-update the kafka-init module due to #14 - so actually it should've been 12 notifications!
• there are more notifications than anyone will ever actually read
• messages for commits (excluding merge) are missing the PR number (we need #8 to fix this)
• the HipChat card titles are not helpful ("The Travis CI build passed") - should really be the repo name, PR number, and title
• we can probably just discard push events (continuous-integration/travis-ci/push context)
• merge messages and follow-ups are semi-duplicates, maybe we could merge these or just drop merge messages?

Minimally I'd like to see the following messages:

• 1 message for initial commit passing
• 1 message per followup commit passing
• 1 message on merge
• 1 message per followup action
  • should actually be informative, not something useless like "GitHub pages finished building"
  • e.g. "auto-update made to ..." or "release published ..."

It would also be nice to use collapsed cards for all non-error messages. Link directly to the PR via the title and keep the details hidden.

For the above example this would cut it down to 8-9 messages. To be honest this is still too many, but I'm not sure how to cut this down more. At least we can make the messages we do send actually helpful so there's a reason to read them.

It would be nice if we could include a summary of commits between versions in auto-generated PRs. Probably just a bulleted list of commit message titles between versions.

latest can be treated as the "recommended promoted artifact", so when possible we should try to use any other (more specific) tags that resolve to the same hash.

Since many tags can refer to the same hash, we'll need some heuristic to decide which is the best / most unique. It may sound dumb, but string length might be a good metric. Otherwise we'd need to use a regex to find timestamps or something.

Per https://issuetracker.google.com/issues/36252545, GCF now provides access to the raw request body, meaning we can securely validate webhooks from GitHub in more or less the same way that we validate for standalone. We can remove the 2nd webhook now.

The bot will make a branch in its fork for every update. These build up quite a bit over time (monasca-ci has 88 old branches now). It would be nice if branches were automatically deleted when a PR is closed.

The bot should leave a warning (maybe as a comment on the pull request) when the existing version doesn't match the fromVersion from the Update instance. In most cases this is probably fine, but it'd be good to know to take a closer look at the patch before applying (in case a dependency isn't recognized quite correctly).

.drone.yml can depend on docker images which we could track and update.

The main complication is that drone projects that use secrets require a .drone.yml.sig which the bot will not be able to update.

• could the bot be configured to update drone signatures? lots of complexity, not much benefit
• is it acceptable to ignore repos that require a signed .drone.yml?
• would there be value in proposing a PR with the changes and letting another user apply the update?
  • this may not save devs much time, since they'd have to clone the branch and sign the changes manually
• could we instead file an issue rather than making a PR?

Right now the bot doesn't care if dependencies stay out of date for a long time. For instance, if a dependency reference is already out of date when the bot is initialized it will never file a PR despite the version mismatch, and if a generated PR is closed or never merged it just assumes that it's up-to-date.

The bot can take a few actions to improve the UX:

1. if dependencies fall out of date for some amount of time (say a week), an issue can be filed on the dependent project to notify devs about the issue (for first-time setup or closed PR)
   • if the issue is then closed the update should be ignored
2. PRs that are never merged should at least be updated as additional updates come in

(depends on #2 to check for updates intermittently and parts of #8 since PRs would actually need to be tracked properly)

The last ... many ... updates to several modules in monasca-helm always show the fromVersion of e.g. kafka as being 0.2.0. For some reason we only generate updates when new versions are actually pushed

I've also confirmed that the database version is stuck at 0.2.0 so it isn't some odd template rendering bug or anything like that.

The bot filed monasca/monasca-helm#182 automatically, but the same upgrade was merged manually in monasca/monasca-helm#183.

We should be able to detect this case and close existing PRs. Presumably requires #8.

A small webpage to show tracked repositories, modules, versions, and dependencies would be very helpful. With authentication it could even be used to manually trigger updates, releases, etc.

It would be trivial to serve a page via the express server. For GCF we'd need a different solution but it would probably just be a static page that makes API calls (so a GCP bucket). We would probably want to remove authentication from read-only endpoints, though. Also, we'll need some CORS rules.

A tool like this would be a big help for debugging issues like #14 and #22.

It would be extremely useful to have some support for repository subsetting. Right now we track all modules in a given repository, but this would be bad behavior for things like e.g. the public Helm repo. It would be nice if we could only check for updates to the modules from these repositories that we care about (or that have some reverse dependency relationship)

Probably depends on #2 since we'll only want to update these repositories every 24 hours or so.

It would be nice to have (in addition to mutation/update plugin) support for release plugin. These would perform whatever tasks are needed to release a new version. Some examples:

• PR to helm git source to bump chart version, user can specify a new version number or 'major', 'minor', or 'patch' to increment the respective field by +1 set others to 0
  • CI in monasca-helm will pick up the change and automatically release once the PR is approved
• PR to docker git source to trigger rebuild
  • for images using dbuild and the master-datestamp versioning policy, just put up an empty commit with !push <module> in the commit message
  • for for semver, follow helm semantics
  • once approved, images are rebuild and pushed to docker hub, pr-bot already notifies about the change

It'd be nice to use the pr bot to send notification on CI/CD pass / fail for pull requests. Unfortunately status events don't include any details about their associated pull request and only have the commit sha. To send notifications properly, we'll need to keep tracked PRs and their hashes in the database.

Realistically this means fleshing out https://github.com/monasca/pr-bot/blob/master/lib/pullrequest.js more fully.

Right now we only support HipChat which is less than ideal.

js-yaml changes the formatting from what helm outputs by default. Pretty minor, but would be nice.

(yawn-yaml will be included already since it's being used for values.yaml modifications)

When available, https://cloud.google.com/pubsub/docs/ should be used to deliver notifications. HipChat servers are often slow to respond and can time out. PubSub would decouple notification failures from an otherwise successful action, and help to ensure delivery since we could then retry.

Also, when we don't need to run shell commands, we can use much smaller functions. A 256 MiB function can run for 60 seconds waiting on a slow HipChat endpoint at 1/8th the cost of our main function.