Code Monkey home page Code Monkey logo

monorepo-diff-buildkite-plugin's Introduction

e2e status codecov Publish

monorepo-diff-buildkite-plugin

This project is archived

I no longer use Buildkite, and unfortunately, I don't have time to maintain this repository.

This plugin will assist you in triggering pipelines by watching folders in your monorepo.

Check out this post to learn How to set up Continuous Integration for monorepo using Buildkite.

Using the plugin

If the version number is not provided then the most recent version of the plugin will be used. Do not use version number as master or any branch names.

Simple

steps:
  - label: "Triggering pipelines"
    plugins:
      - monebag/monorepo-diff#v2.5.9:
          diff: "git diff --name-only HEAD~1"
          watch:
            - path: "bar-service/"
              config:
                command: "echo deploy-bar"
            - path: "foo-service/"
              config:
                trigger: "deploy-foo-service"

Detailed

steps:
  - label: "Triggering pipelines"
    plugins:
      - monebag/monorepo-diff#v2.5.9:
          diff: "git diff --name-only $(head -n 1 last_successful_build)"
          interpolation: false
          env:
            - env1=env-1 # this will be appended to all env configuration
          hooks:
            - command: "echo $(git rev-parse HEAD) > last_successful_build"
          notify:
            - email: [email protected]
            - basecamp_campfire: https://basecamp-url
            - webhook: https://webhook-url
            - pagerduty_change_event: 636d22Yourc0418Key3b49eee3e8
            - github_commit_status:
                context: my-custom-status
            - slack: '@someuser'
              if: build.state === "passed"
          watch:
            - path:
                - "ops/terraform/"
                - "ops/templates/terraform/"
              config:
                command: "buildkite-agent pipeline upload ops/.buildkite/pipeline.yml"
                label: "Upload pipeline"
                # following configs are available in command. notify is not available in trigger step
                notify:
                  - basecamp_campfire: https://basecamp-url
                  - github_commit_status:
                      context: my-custom-status
                  - slack: '@someuser'
                    if: build.state === "passed"
                # soft_fail: true
                soft_fail:
                  - exit_status: 1
                  - exit_status: "255"
                retry:
                  automatic:
                  - limit: 2
                    exit_status: -1
                agents:
                  queue: performance
                artifacts:
                  - "logs/*"
                env:
                  - FOO=bar
            - path: "foo-service/"
              config:
                trigger: "deploy-foo-service"
                label: "Triggered deploy"
                build:
                  message: "Deploying foo service"
                  meta_data:
                    build_number: "123"
                  env:
                    - HELLO=123
                    - AWS_REGION

          wait: true

Configuration

diff (optional)

This will run the script provided to determine the folder changes. Depending on your use case, you may want to determine the point where the branch occurs https://stackoverflow.com/questions/1527234/finding-a-branch-point-with-git and perform a diff against the branch point.

Sample output:

README.md
lib/trigger.bash
tests/trigger.bats

Default: git diff --name-only HEAD~1

Examples:

diff: ./diff-against-last-successful-build.sh

#!/bin/bash

set -ueo pipefail

LAST_SUCCESSFUL_BUILD_COMMIT="$(aws s3 cp "${S3_LAST_SUCCESSFUL_BUILD_COMMIT_PATH}" - | head -n 1)"
git diff --name-only "$LAST_SUCCESSFUL_BUILD_COMMIT"

diff: ./diff-against-last-built-tag.sh

#!/bin/bash

set -ueo pipefail

LATEST_BUILT_TAG=$(git describe --tags --match foo-service-* --abbrev=0)
git diff --name-only "$LATEST_TAG"

interpolation (optional)

This controls the pipeline interpolation on upload, and defaults to true. If set to false it adds --no-interpolation to the buildkite pipeline upload, to avoid trying to interpolate the commit message, which can cause failures.

env (optional)

The object values provided in this configuration will be appended to env property of all steps or commands.

log_level (optional)

Add log_level property to set the log level. Supported log levels are debug and info. Defaults to info.

steps:
  - label: "Triggering pipelines"
    plugins:
      - monebag/monorepo-diff#v2.5.9:
          diff: "git diff --name-only HEAD~1"
          log_level: "debug" # defaults to "info"
          watch:
            - path: "foo-service/"
              config:
                trigger: "deploy-foo-service"

watch

Declare a list of

- path: app/cms/
  config: # Required [trigger step configuration]
    trigger: cms-deploy # Required [trigger pipeline slug]
- path:
    - services/email
    - assets/images/email
  config:
    trigger: email-deploy

path

If the path specified here in the appears in the diff output, a trigger step will be added to the dynamically generated pipeline.yaml

A list of paths can be provided to trigger the desired pipeline. Changes in any of the paths will initiate the pipeline provided in trigger.

A path can also be a glob pattern. For example specify path: "**/*.md" to match all markdown files.

config

Configuration supports 2 different step types.

Trigger

The configuration for the trigger step https://buildkite.com/docs/pipelines/trigger-step

By default, it will pass the following values to the build attributes unless an alternative values are provided

- path: app/cms/
  config:
    trigger: cms-deploy
    build:
      commit: $BUILDKITE_COMMIT
      branch: $BUILDKITE_BRANCH

wait (optional)

Default: true

By setting wait to true, the build will wait until the triggered pipeline builds are successful before proceeding

hooks (optional)

Currently supports a list of commands you wish to execute after the watched pipelines have been triggered

hooks:
  - command: upload unit tests reports
  - command: echo success

Command

steps:
  - label: "Triggering pipelines"
    plugins:
      - monebag/monorepo-diff#v2.5.9:
          diff: "git diff --name-only HEAD~1"
          watch:
            - path: app/cms/
              config:
                group: ":rocket: deployment"
                command: "netlify --production deploy"
                label: ":netlify: Deploy to production"
                agents:
                  queue: "deploy"
                env:
                  - FOO=bar

There is currently limited support for command configuration. Only the command property can be provided at this point in time.

Using commands, it is also possible to use this to upload other pipeline definitions

- path: frontend/
  config:
    command: "buildkite-agent pipeline upload ./frontend/.buildkite/pipeline.yaml"
- path: infrastructure/
  config:
    command: "buildkite-agent pipeline upload ./infrastructure/.buildkite/pipeline.yaml"
- path: backend/
  config:
    command: "buildkite-agent pipeline upload ./backend/.buildkite/pipeline.yaml"

How to Contribute

Please read contributing guide.

monorepo-diff-buildkite-plugin's People

Contributors

adikari avatar allcontributors[bot] avatar chronotc avatar elliott-davis avatar glittershark avatar harrietgrace avatar isaacsu avatar jonathanlorimer avatar jquick avatar jullianoacqio avatar lcolman-seek avatar lstoll avatar nhurden avatar pmenglund avatar uniquefine avatar worxli avatar xzyfer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

monorepo-diff-buildkite-plugin's Issues

Update plugin version in README examples

The watch feature for multiple paths & trigger single pipeline was introduced in 1.1.0 while the examples in the README show the feature availability for v1.0.0 which is not true. It took me a while to figure this out.

add examples with separate readme

Provide full config and script

  • examples/with-github-compare-api/README.md
  • examples/with-package-json-version-changes/README.md
  • examples/with-e2e-test-as-hooks/README.md

How to use `hooks` with a specific agent selection rule

I'm using this configuration

---
steps:
  - label: "Triggering pipelines :rocket:"
    plugins:
      - chronotc/monorepo-diff#v1.0.0:
          diff: "./script/modified-files.sh"
          watch:
            - path: "packages/api"
              config:
                trigger: "api"
            - path: "packages/infra"
              config:
                trigger: "infra"
          wait: true
          hooks:
            - command: echo "every that had to build is complete"
    agents:
      queue: custom-agents

This config specifies a wait and a hooks option.

I'm seeing two things which are not what I expect:

  1. hooks command is started in a separate build step as soon as the diff is complete. That means the wait command is being ignored.
  2. the build steps for the hooks commands ignore the specified agent queue and instead set the queue to default which means no agent will pick the step up:

image

Am I missing something or is there a way to handle the above two points?

Thanks 🙂

FATA[2022-12-06T22:43:41Z] diff command failed: command `git` failed: exit status 128

Using the example in your README throws an error:

steps:
  - label: "Unit Test Pipeline"
    plugins:
      - monebag/monorepo-diff#v2.5.0:
          diff: "git diff --name-only $(head -n 1 last_successful_build)"
          hooks:
            - command: "echo $(git rev-parse HEAD) > last_successful_build"
          watch:
            - path:
                - "applications/react/modern/feature-administration/**"
              config:
                - label: "Feature Administration"
                  <<: *default_config
                  commands:
                    - "make titan-feature-administration"
                    - "make test-titan-feature-administration"
            - path:
                - "applications/react/legacy/temporary-access/**"
              config:
                - label: "Temporary Access"
                  <<: *default_config
                  commands:
                    - "make titan-temporary-access"
                    - "make test-titan-temporary-access"
            - path: 
                - "applications/react/modern/template/**"
              config:
                - label: "Template"
                  <<: *default_config
                  commands:
                    - "make titan-template"
                    - "make test-titan-template"
            - path:
                - "applications/react/modern/login-hero/**"
              config:
                - label: "Login Hero"
                  <<: *default_config
                  commands:
                    - "make titan-login-hero"
                    - "make test-titan-login-hero"
            - path:
                -"applications/react/legacy/legacy-template/**"
              config:
                - label: "Legacy Template"
                  <<: *default_config
                  commands:
                    - "make titan-legacy-template"
                    - "make test-titan-legacy-template"
            - path:
                - "applications/react/legacy/invoca-tags/**"
              config:
                - label: "Invoca Tags"
                  <<: *default_config
                  commands:
                    - "make titan-invoca-tags"
                    - "make test-titan-invoca-tags"
            - path:
                - "applications/react/modern/ivr-campaigns/**"
              config:
                - label: "IVR Campaigns"
                  <<: *default_config
                  commands:
                    - "make titan-ivr-campaigns"
                    - "make test-titan-ivr-campaigns"
            - path:
                - "applications/react/legacy/partner-platforms/**"
              config:
                - label: "Partner Platforms"  
                  <<: *default_config
                  commands:
                    - "make titan-partner-platforms"
                    - "make test-titan-partner-platforms"
            - path:
                - "applications/react/legacy/bulk-number-management/**"
              config:
                - label: "Bulk Number Management"
                  <<: *default_config
                  commands:
                    - "make titan-bulk-number-management"
                    - "make test-titan-bulk-number-management"
            - path:
                - "applications/react/legacy/customer-requests/**"
              config:
                <<: *default_config
                commands:
                  - "make titan-customer-requests"
                  - "make test-titan-customer-requests"
            - path:
                - "applications/react/modern/active-ai/**"
              config:
                <<: *default_config
                commands:
                    - "make titan-active-ai"
                    - "make test-titan-active-ai"
            - path:
                - "applications/react/modern/signals/**"
              config:
                - label: "Signals"
                  <<: *default_config
                  commands:
                      - "make titan-signals"
                      - "make test-titan-signals"

Shows:

INFO[0000] --- :one: monorepo-diff v2.5.0
--
  | INFO[2022-12-06T22:43:41Z] Running diff command: git diff --name-only $(head -n 1 last_successful_build)
  | FATA[2022-12-06T22:43:41Z] diff command failed: command `git` failed: exit status 128
  | 🚨 Error: The command exited with status 1

How to allow a triggered pipeline to fail?

I'm triggering a pipeline with custom env variables that I would like to allow to soft_fail so it doesn't fail my parent pipeline. Is there a way to do that?

Thanks in advance!

Provide arm64 binary

ARM-based Buildkite CI/CD has become very popular. Wondering if the team could release linux-arm64 binary?

Many thanks

Add common path in config

Let's say we have a few common paths like gen files, lib/package, and other dependencies. In the current setup, we need to add them to all the services but if we have a common config then we just need to add it at one place

example

steps:
  - label: "Triggering pipelines"
    plugins:
      - chronotc/monorepo-diff#v2.3.0:
          diff: "git diff --name-only $(head -n 1 last_successful_build)"
          interpolation: false
          env:
            - env1=env-1 # this will be appended to all env configuration
          hooks:
            - command: "echo $(git rev-parse HEAD) > last_successful_build"
          common:
                - "gen/"
                - "api/"
          watch:
            - path:
                - "ops/terraform/"
                - "ops/templates/terraform/"
              config:
                command: "buildkite-agent pipeline upload ops/.buildkite/pipeline.yml"
                label: "Upload pipeline"
                retry:
                  automatic:
                  - limit: 2
                    exit_status: -1
                agents:
                  queue: performance
                artifacts:
                  - "logs/*"
                env:
                  - FOO=bar
            - path: 
                - "foo-service/"
              config:
                trigger: "deploy-foo-service"
                label: "Triggered deploy"
                build:
                  message: "Deploying foo service"
                  env:
                    - HELLO=123
                    - AWS_REGION
          wait: true

Is there a way to invoke pipelines/command one by one when multiple changes happened in multiple path?

steps:
  - label: "Detecting which mono-repo has changed"
    plugins:
      - chronotc/monorepo-diff#v1.3.0:
          diff: "git diff --name-only HEAD~1"
          watch:
            - path: "folder-a/"
              config:
                label: ":pipeline: Continue blue-alice pipeline"
                command: "buildkite-agent pipeline upload ./folder-a/.buildkite/pipeline.yml"
            - path: "folder-b/"
              config:
                label: ":pipeline: Continue dwh-store pipeline"
                command: "buildkite-agent pipeline upload ./folder-b/.buildkite/pipeline.yml"

Hey guys, my root directory's pipeline.yml looks like that. I have sub-directories having it's own defined pipeline.ymland all invoked from this root directory's pipeline.yml.

Is there a way that I can ensure each pipeline run in order? Currently I tried concurrency group and - wait in each sub-directoriy's pipeline.yml but no luck.

Would really appreciate your thinking and replies!

Group Merging Support

I'd like to use group merging to merge the monorepo step that triggers my steps into one buildkite tile in the UI.

Thus I'd like to be able to add a group in my config

            - path:
                - "**/*.js"
              config:
                group: prettier
                label: ":sparkles: :nail_care: Prettier"
                command: "buildkite-agent pipeline upload ./.buildkite/prettier.yml"

So that it would merge with my uploaded pipeline:

steps:
  - group: prettier
    steps:
      - label: ":sparkles: :nail_care: Prettier"
        command:
          - prettier

An alternative, and perhaps more correct config structure would be:

            - path:
                - "**/*.js"
              config:
                group: prettier
                steps:
                  - label: ":sparkles: :nail_care: Prettier"
                    command: "buildkite-agent pipeline upload ./.buildkite/prettier.yml"

Update default diff checking logic?

I haven't been using this plugin too extensively, but I was curious if the git diff logic could have a slightly more sensible default.

In general, I think open PRs (or branches) should always diff against master (or default branch) since you might not trigger a required pipeline on every commit.

For example:

  • You make a huge new feature
  • The tests fail
  • You edit README.md that doesn't trigger any pipelines
  • You get a green build since no tests were run.

This seems like a pitfall a ton of users can fall into.

Something like this might be a good alternative?

if [ "$BUILDKITE_BRANCH" = "$BUILDKITE_PIPELINE_DEFAULT_BRANCH" ]; then
  git diff --name-only HEAD~1
else
  git diff --name-only "origin/$BUILDKITE_PIPELINE_DEFAULT_BRANCH"
fi

It's easy enough to drop a custom script in here, but food for thought!

Use wildcard in path

I'd like to be able to set a "default" pipeline to be run if no other watch paths conditions are met.

Example of how the configuration would look like:

      - chronotc/monorepo-diff#v1.1.1:
          watch:
            - path: "packer/"
              config:
                trigger: "packer"
            - path:
                - "*"
              config:
                trigger: "terraform"
          wait: true

add support for artifacts

It should be possible to collect artifacts from a command triggered by the monorepo diff plugin:

plugins:
  - chronotc/monorepo-diff#v1.3.0:
      watch:
      - path:
        - tests/X
        config:
          command: "run test X"
          artifact_paths:
          - logs/tests/*

Error: command `git` failed: exit status 128

Hi, i'm dealing with this problem for hours now so appreciate some comments here. I'm using something very similar to the default detailed template provided in the README. The problem is that whenever I try to trigger it using a buildkite setup, I'm facing this error:

$ /etc/buildkite-agent/plugins/github-com-chronotc-monorepo-diff-buildkite-plugin-v2-0-4/hooks/command
https://github.com/chronotc/monorepo-diff-buildkite-plugin/releases/download/v2.0.4/monorepo-diff-buildkite-plugin-linux
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   644  100   644    0     0   5750      0 --:--:-- --:--:-- --:--:--  5750
100 3302k  100 3302k    0     0  11.6M      0 --:--:-- --:--:-- --:--:-- 11.6M
INFO[0000] --- :one: monorepo-diff v2.0.4
INFO[2021-06-04T10:25:40Z] Running diff command: git diff --name-only $(head -n 1 ~/last_successful_build)
FATA[2021-06-04T10:25:40Z] diff command failed: command `git` failed: exit status 128
🚨 Error: The command exited with status 1

Here is my pipeline.yml and if I change the diff to git diff --name-only HEAD~1 it is working without error.
I can aslo confirm that the file last_successful_release exist, contains the hash of the branch point and running the command git diff --name-only $(head -n 1 ~/last_successful_build) in the server (using buildkite user) works perfectly fine.

steps:
  - label: "Triggering pipelines"
    plugins:
      - chronotc/monorepo-diff#v2.0.4:
          # here is the problem, more specifically the lines inside the $()
          diff: "git diff --name-only $(head -n 1 ~/last_successful_build)"
          interpolation: false
          watch:
            - path:
                - "repo1/"
              config:
                command: "buildkite-agent pipeline upload repo1/.buildkite/pipeline.yml"
                label: "repo1"
            - path:
                - "repo2/"
              config:
                command: "buildkite-agent pipeline upload repo2/.buildkite/pipeline.yml"
                label: "repo2"
            - path:
                - "repo3/"
              config:
                command: "buildkite-agent pipeline upload repo3/.buildkite/pipeline.yml"
                label: "repo3"
  - wait
  - label: "Mark Build"
    # This is working fine
    command: "echo $(git rev-parse HEAD) > ~/last_successful_build"

Generate pipeline is not loading in v2.0.4

Using the version v2.0.4 of this plugin, the generated build is output into console but not load

~~~ Running plugin monorepo-diff command hook
[2021-05-06T23:25:25Z] �[90m$�[0m /var/lib/buildkite-agent/plugins/github-com-chronotc-monorepo-diff-buildkite-plugin-v2-0-4/hooks/command
[2021-05-06T23:25:25Z] https://github.com/chronotc/monorepo-diff-buildkite-plugin/releases/download/v2.0.4/monorepo-diff-buildkite-plugin-linux
[2021-05-06T23:25:25Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[2021-05-06T23:25:25Z]                                  Dload  Upload   Total   Spent    Left  Speed
[2021-05-06T23:25:25Z] 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   644  100   644    0     0   3833      0 --:--:-- --:--:-- --:--:--  3856
[2021-05-06T23:25:26Z] 
100 3302k  100 3302k    0     0  5449k      0 --:--:-- --:--:-- --:--:-- 5449k
[2021-05-06T23:25:26Z] �[36mINFO�[0m[0000] --- :one: monorepo-diff v2.0.4               
[2021-05-06T23:25:26Z] �[36mINFO�[0m[2021-05-06T23:25:26Z] Running diff command: .buildkite/diff.bash   
[2021-05-06T23:25:26Z] Generated Pipeline:
[2021-05-06T23:25:26Z] steps:
[2021-05-06T23:25:26Z] - trigger: webhooks
[2021-05-06T23:25:26Z]   build:
[2021-05-06T23:25:26Z]     message: Why no work
[2021-05-06T23:25:26Z]     branch: API-871
[2021-05-06T23:25:26Z]     commit: 3fa1531e2cfd976584ee5ce8bc3dde83ae2023b9
[2021-05-06T23:25:26Z] - {}
[2021-05-06T23:25:26Z] - wait

in the timeline of the step, it says it uploaded the steps:

image

however, none of the steps are uploaded or run. when I switch back to version v1.3.1 of this plugin, everything works as expected with no other changes.

Binary download URL being populated with wrong architecture on Apple Silicon M1 Mac

Bug: all $(arch)'s other than aarch64 are assumed to be amd64

Issue observed on MacBook Pro (16-inch, 2021) running Apple M1 Pro chip with macOS Monterey when running the buildkite agent locally.

This is causing the plugin to assemble and download the binary for the wrong architecture. In my case (on arm64) it is getting: monorepo-diff-buildkite-plugin-linux-amd64.
This then results in an error similar to the following:

/var/folders/p5/2smnxkqs3fv27_5g43jg06_q0000rq/T/buildkite-plugins-906434519/github-com-chronotc-monorepo-diff-buildkite-plugin-v2-1-2/hooks/command:
line 27: ./monorepo-diff-buildkite-plugin-linux-amd64: cannot execute binary file

With "cannot execute binary file" here being the error that the ELF binary is for the wrong machine architecture.

Minimum Steps to Reproduce

The following pipeline.yml logic is sufficient to cause the error (when run on an M1 Mac using local agent).

steps:
  - plugins:
      - chronotc/monorepo-diff#v2.1.2:
          watch:
            - path: "/dummy-value"
              config:
                command: "arch"

Different possible values for the architecture

On Apple Silicon

Output of arch on my system:

arm64

Output of uname -mps on my system:

Darwin arm64 arm

In the wild

According to some answers on StackOverflow, as well as on Wikipedia; there are many values for the system architecture. Even just for the ARM and x86_64 / amd64 instruction sets.
At a minimum, I would think the amd64, x86_64, aarch64 and arm64 values from arch (or uname -p) should be handled correctly to ensure trouble free adoption of the plugin.

Add support for commands

Currently this plugin only supports trigger steps.
May be useful to support command steps as well......

could not initialize plugin

Hi the plugin suddenly stopped working for us

$ /var/lib/buildkite-agent/plugins/github-com-chronotc-monorepo-diff-buildkite-plugin/hooks/command

  | https://github.com/monebag/monorepo-diff-buildkite-plugin/releases/latest/download/monorepo-diff-buildkite-plugin-linux-arm64
  | % Total % Received % Xferd Average Speed Time Time Time Current
  | Dload Upload Total Spent Left Speed
  | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
  | 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
  | 100 3244k 100 3244k 0 0 3903k 0 --:--:-- --:--:-- --:--:-- 3903k
  | INFO[0000] --- 1️⃣ monorepo-diff v2.5.0
  | FATA[0000] could not initialize plugin

commit messages can trigger interpolation errors

If the commit message contains something like:

A test change

Testcase: how to break
```
$ command line example
```

This will fail to run, as buildkite pipeline upload tries to interpolate the pipeline config the monorepo-diff generates.

The quick fix is to just add --no-interpolation to https://github.com/chronotc/monorepo-diff-buildkite-plugin/blob/408ce9c633eb532dad3a914c3c8336025b82f942/hooks/command#L29 but I don't know if someone depends on it being interpolated, so a better solution would be to add

    plugins:
      - chronotc/monorepo-diff#v1.3.0:
          no_interpolation: true
          watch:
            ...

so it is optional.

'last_sucessful_build' - how is this seeded into a build-node working copy?

My attempts to try this out fail because the build-node that BK allocates doesn't have a last_sucessful_build file in it. There's nowhere in the YAML config that details what to do if last_sucessful_build isn't found.

Do I have to SSH to multiple build agents to create that file?

Or do I check-in some bash fu that checks for the existence of that file, and it it's absence makes a default version?

Support setting agent queue when running commands?

Would it be possible to add a way to specify an agent queue for the steps run by this plugin?

Ex:

steps:
  - label: ":github:"
    plugins:
      - chronotc/monorepo-diff#v1.2.0:
          watch:
            - path: "dir1/"
              config:
                command: "echo hello"
                agent:
                  queues: ops

Permission denied when executing custom diff script

I'm getting permission denied when attempting to specify a custom diff script.

/usr/local/var/buildkite-agent/plugins/github-com-chronotc-monorepo-diff-buildkite-plugin-v1-1-1/lib/diff.bash: line 13: ./scripts/custom_diff.sh: Permission denied
Failed to run diff command

I've tried adding two commands in the "diff:" yml section (chmod +x ... ; custom.sh ...) but it doesn't like it:

Running diff command: [chmod +x ./scripts/custom_diff.sh ; ./scripts/custom_diff.sh cb1ddeaed62004ad62712a89475407bdbdfbea56]
--
  | chmod: ;: No such file or directory
  | chmod: cb1ddeaed62004ad62712a89475407bdbdfbea56: No such file or directory

Pipelines with a slash don't work

Looks like either this plugin (or maybe Buildkite's API) doesn't support triggering a pipeline that has a / in the name (ie. foo/bar). You get a 422 error:


2020-06-19 23:40:37 ERROR  Unrecoverable error, skipping retries
--
  | 2020-06-19 23:40:37 FATAL  Failed to upload and process pipeline: POST https://agent.buildkite.com/v3/jobs/44176033-bf4f-4ca9-8370-d90809786225/pipelines: 422 Pipeline "foo/bar" does not exist
  | 🚨 Error: The command exited with status 1

My pipeline is defined as:

steps:
  - label: 'Triggering pipelines'
    plugins:
      - chronotc/monorepo-diff#v1.1.1:
          watch:
            - path: 'packages/something'
              config:
                trigger: 'foo/bar'

Removing the slash from the name of the pipeline makes it work.

rewrite with go

It's not unmaintainable but hopefully it much more readable with go.

Bug: Failure to upload generated pipeline doesn't fail the job

I had an improperly configured job definition that was causing the monorepo-diff plugin to generate an invalid pipeline, which failed at the upload step. However, the overall job was marked as a success.

Skimming the code, I think it is because nothing is done with the return value of generatePipeline here.

Here is the debug output of the job in question:

Running plugin monorepo-diff command hook
  | $ /var/lib/buildkite-agent/plugins/github-com-chronotc-monorepo-diff-buildkite-plugin-v2-0-4/hooks/command
  | https://github.com/chronotc/monorepo-diff-buildkite-plugin/releases/download/v2.0.4/monorepo-diff-buildkite-plugin-linux
  | % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
  | Dload  Upload   Total   Spent    Left  Speed
  | 100   644  100   644    0     0  35777      0 --:--:-- --:--:-- --:--:-- 37882
  | 100 3302k  100 3302k    0     0  48.1M      0 --:--:-- --:--:-- --:--:-- 48.1M
  | INFO[0000] --- :one: monorepo-diff v2.0.4
  | INFO[2021-07-19T18:14:16Z] Running diff command: .buildkite/shared/scripts/diff.sh
  | DEBU[2021-07-19T18:14:16Z] Output from diff:
  | .buildkite/pipeline.merge.yml
  | Generated Pipeline:
  | steps:
  | - {}
  |  
  | DEBU[2021-07-19T18:14:16Z]
  | command = 'buildkite-agent',
  | args = '[pipeline upload /tmp/bmrd-252730796]',
  | error = '2021-07-19 18:14:16 INFO   Reading pipeline config from "/tmp/bmrd-252730796"
  | 2021-07-19 18:14:16 INFO   Updating BUILDKITE_COMMIT to "<REDACTED>"
  | 2021-07-19 18:14:16 WARN   POST https://agent.buildkite.com/v3/jobs/<REDACTED>/pipelines: 422 Step is missing a type `{}` (Attempt 1/60 Retrying in 5s)
  | 2021-07-19 18:14:16 ERROR  Unrecoverable error, skipping retries
  | 2021-07-19 18:14:16 FATAL  Failed to upload and process pipeline: POST https://agent.buildkite.com/v3/jobs/<REDACTED>/pipelines: 422 Step is missing a type `{}`
  | '


ERROR: unsupported architecture "amd64"

WHAT:
Tried to run the plugin on Buildkite agent on a FreeBSD. The agent works for all Buildkite commands. Returns above error message.

CONFIG:

  • FreeBSD on a x64 AMD CPU
  • OS: 13.1-RELEASE
  • Shell: Bash

PR-related environment variables not passed down.

This is likely a buildkite issue, but it'd be good to know if anyone else is seeing this.

I have the usual:

  • Webhook triggers pipeline
  • Pipeline has single step, which is using this plugin
  • Plugin starts 1 or more other pipelines

If I look at the environment of the first step, I can see that BUILDKITE_PULL_REQUEST and a few others are set. Yay. Expected.
If I look at the "1 or more" others, it is not - its set to false for no PR.

Likely also

BUILDKITE_PULL_REQUEST_BASE_BRANCH
BUILDKITE_PULL_REQUEST_REPO

I'd EXPECT buildkite pipeline upload.... to look at it, and carry them across, but it appears it doesn't. Anyone else see this, or have a work around? I need that specific variable name for the tool (DangerJs) I'm using, and likely other stuff too.

README confusion for "command"

image

Command is a space-separated English sentence there. I'd have guessed that should be an installed command like make <args>, or ./myBuildScript.sh so am confused as to what I learn from what is in the screen-shot.

Indeed, the simple and more complex examples inline atop the README, don't have command nodes in the YAML, just trigger.

Add example for handling "Unable to trigger builds from this pipeline -- no Buildkite user associated" error

Hey there,

A running version of this pipeline was finally established. Unfortunately, I was immediately faced with a failure containing this message:

2019-10-04 19:33:57 FATAL  Failed to upload and process pipeline: POST https://agent.buildkite.com/v3/jobs/dc24201f-b239-4dd9-87c3-aeeb82c72a21/pipelines: 422 Builds cannot be triggered from this pipeline as there is no Buildkite user linked to this build. Ensure that a user has added "[email protected]" as a verified email address on their account.

I took a look at the difference in ENV between this step and the one that triggered it without issue... they both say the same thing, so I'm not sure what the problem is:

BUILDKITE_BUILD_CREATOR="jwarner112"
BUILDKITE_BUILD_CREATOR_EMAIL="[email protected]"
BUILDKITE_BUILD_CREATOR_TEAMS=""

One way of handling this is to do as the message asks. However I've got rather a lot of people that I would have to get to do this, and nobody has ever encountered this before. Have you? If so, would you add an example of facing this issue and your workaround?

(I realize this could be handled at multiple levels, this level just has the smallest scope and I'm hoping you have a workaround)

Support Multiple Triggers

It's common for a monorepo to have many internal dependencies and whenever a change is detected in the dependency, it can be desirable to run the build of it's dependents as well to ensure no breaking changes have been introduced. Obviously this can be circumvented with contract testing, but we aren't there just yet.

Adding list support to the trigger configuration would enable this functionality.

            - path:
                - << *common_watch_paths
                - 'services/newsletter'
                - '.buildkite/pipelines/newsletter'
              config:
                trigger: 
                    - 'newsletter'
                    - 'website'
                build: { message: '${BUILDKITE_MESSAGE}', commit: '${BUILDKITE_COMMIT}', branch: '${BUILDKITE_BRANCH}' }

Add support for globs in paths

It would be useful to trigger certain pipeline only for specific file types. E.g. the markdown linting command/pipeline only for changes in files that end in .md. For this it would be useful to have glob support in paths.

Proposal:

steps:
  - label: "Triggering pipelines"
    plugins:
      - chronotc/monorepo-diff#v2.0.3:
          watch:
            - path: "**/*.md"
              config:
                command: "lint markdown"

Support other distributions

I am unable to run my pipeline locally with the new Go builds since it is only built for Linux. It would be nice if the releases were cross-compiled and the plugin detected which needed to be loaded.

State of project

Hi @chronotc,

Thanks for the great work on this very useful plugin.

Unfortunately I have a few issues with the plugin, which would already be addressed by PRs. Specifically I think about

But there are others as well. It seems none of them got any feedback.
Are you still maintaining this project?
Otherwise it would be cool to have some indication on the repo that you are looking for maintainers, or that a fork should be created.

Thanks a lot for any feedback.

diff command failed: command `git` failed: exit status 128

I ran into this error during my configuration:

- label: ":white_check_mark: Test Coverage"
  timeout_in_minutes: 10
  agents:
    queue: ruby-2-7
    ruby: 2.7
  plugins:
  - monebag/monorepo-diff#v2.5.0:
      diff: git diff --name-only HEAD $(git merge-base HEAD origin/master)
      watch:
        - path:
            - "test/"
            - "client/"
            - "app/"
            - "components/"
            - "components/"
            - "bin/"
          config:
            command: ruby bin/ci_pipeline/test_coverage.rb

Error:

INFO[0000] --- :one: monorepo-diff v2.5.0
INFO[2022-11-15T17:49:52Z] Running diff command: git diff --name-only HEAD $(git merge-base HEAD origin/master)
FATA[2022-11-15T17:49:52Z] diff command failed: command `git` failed: exit status 128
🚨 Error: The command exited with status 1

Thanks for any help!

[Help] Runner shows the outgoing steps but doesn't actually run them?

I have a wokflow that looks like this:

- label: ":coffee: Legacy Client Tests"
  timeout_in_minutes: 60
  env:
    NODE_VERSION: 12.22.7
    GRUNT_QUIET: 'true'
  agents:
    queue: client-tests-ruby-2-7
    ruby: 2.7
  plugins:
  - improbable-eng/metahook#v0.4.1:
      pre-command: ./bin/ci_pipeline/client_test_setup.sh
  - monebag/monorepo-diff#v2.5.0:
      diff: "git diff --name-only HEAD~1"
      watch:
        - path:
          - "yarn.lock"
          - "app/"
          - "backdraft/"
          - "client/"
          - "components/"
          - "test/"
          config:
            command: grunt coffee-test
            agents:
              queue: client-tests-ruby-2-7
              ruby: 2.7

Unfortunately when it runs, it prints this:

$ /buildkite/plugins/github-com-monebag-monorepo-diff-buildkite-plugin-v2-5-0/hooks/command
--
  | https://github.com/monebag/monorepo-diff-buildkite-plugin/releases/download/v2.5.0/monorepo-diff-buildkite-plugin-linux-amd64
  | % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
  | Dload  Upload   Total   Spent    Left  Speed
  | 0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  | 100 3293k  100 3293k    0     0  27.1M      0 --:--:-- --:--:-- --:--:--  139M
  | INFO[0000] --- :one: monorepo-diff v2.5.0
  | INFO[2022-11-18T11:44:48-08:00] Running diff command: git diff --name-only HEAD~1
  | Generated Pipeline:
  | steps:
  | - command: grunt coffee-test
  | agents:
  | queue: client-tests-ruby-2-7
  | ruby: ""

I see the command I want to run there, grunt coffee-test but the command never runs.

  | Generated Pipeline:
  | steps:
  | - command: grunt coffee-test
  | agents:
  | queue: client-tests-ruby-2-7
  | ruby: ""
  |  
  | Running global post-command hook | 0s
  | ⚠️ Warning: Value of DATABASE_MYSQL_PASSWORD below minimum length and will not be redacted
  | ⚠️ Warning: Value of BUNDLE_SILENCE_ROOT_WARNING below minimum length and will not be redacted
  | $ /buildkite/hooks/post-command
  | Running plugin metahook post-command hook

Question/Feature Request: Get current path?

Hi there. This is a question and potentially feature request if it's not currently possible and other people are interested. I see that the contribution guidelines ask that we start a discussion first, however I can't see the Discussions tab so apologies for writing in the wrong place.

What I'd like to do is, inside a child pipeline, refer to the name of the path used to trigger that pipeline. This would allow the child pipeline to be aware of which directory in the monorepo it belongs to. Here is my usecase:

# Repo Structure

.buildkite/
    pipeline.yml
    build.sh
    lint.sh
    test.sh
project-a/
    .buildkite/
        pipeline.yml
project-b/
    .buildkite/
        pipeline.yml
# .buildkite/pipeline.yml

steps:
  - label: ":arrow_right:"
    plugins:
      - chronotc/monorepo-diff#v2.0.4:
          diff: "git diff --name-only HEAD~1"
          watch:
            - path: "project-a/"
              config:
                trigger: project-a
            - path: "project-b/"
              config:
                trigger: project-b
# project-a/.buildkite/pipeline.yml

steps:
  -  command: "./.buildkite/lint.sh project-a"
  -  command: "./.buildkite/build.sh project-a"
# project-b/.buildkite/pipeline.yml

steps:
  -  command: "./.buildkite/lint.sh project-b"
  -  command: "./.buildkite/test.sh project-b"

Inside each .sh file, I am cding into the directory, in this case project-a or project-b in order to run the given task for that step. This is passed as a parameter as in command: "./.buildkite/lint.sh project-a". eg. the lint.sh file contains:

# .buildkite/lint.sh

...
cd $1
npm i
npm run lint
cd -

Instead of passing the directory as a parameter to the scripts, I'd like to be able to handle this programmatically. An environment variable would do fine and would fit the Buildkite pattern for accessing meta-level information but anything would do really.

This could be accessible within the script, eg.

# .buildkite/lint.sh

...
cd $MONOREPO_DIFF_PATH
npm i
npm run lint
cd -

Or kept as a script parameter but accessible from the child pipeline config, eg.

# project-a/.buildkite/pipeline.yml

command: "./.buildkite/lint.sh $MONOREPO_DIFF_PATH"

Alternative Options

Use current directory

This doesn't work because the child pipelines are still being run at the top level of the repo so pwd won't help, and $0 doesn't help because the scripts live in the parent .buildkite directory.

Use pipeline slug

I happen to be using a pattern where the project directories are a match with the pipeline slugs, however this is really more for clarity than utility. I'm very reluctant to start coupling pipeline slugs directly with directory names for maintainability purposes.

Of the options available, I consider my least worst option to be hardcoding the directory names in each child pipeline.yml file and passing them into the scripts, as I'm already doing. It's very prone to human error, and I've already found myself making copy-pasting mistakes just with two child pipelines which cause me to run the wrong thing. That kind of error might go unnoticed for a long time, so I'd like to avoid it entirely. Is there a better way I'm not seeing, or could it be possible to add a feature to the plugin which allows me to interrogate the context in this way? Thanks for reading.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.