buildkite / buildkite-signed-pipeline Goto Github PK

View Code? Open in Web Editor NEW

27.0 22.0 2.0 91 KB

[Deprecated] This is a tool that adds some extra security guarantees around Buildkite's jobs

License: BSD 3-Clause "New" or "Revised" License

Go 97.08% Makefile 2.92%

buildkite buildkite-plugin

buildkite-signed-pipeline's Introduction

buildkite-signed-pipeline

Important

Signed pipelines has been built into the Buildkite agent itself making this utility unnecessary. For more information, see the documentation on Signed pipelines.

This is a tool that adds some extra security guarantees around Buildkite's jobs. Buildkite security best practices suggest using --no-command-eval which will only allow local scripts in a checked out repository to be run, preventing arbitrary commands being injected by an intermediary.

The downside of that approach is that it also comes with the recommendation of disabling plugins, or allow listing specifically what plugins and parameters are allowed. This tool is a collaboration between SEEK and Buildkite that attempts to bridge this gap and allow uploaded steps to be signed with a secret shared by all agents, so that plugins can run without any concerns of tampering by third-parties.

Example

Uploading a pipeline with signatures

Upload is a thin wrapper around buildkite-agent pipeline upload that adds the required signatures. It behaves much like the command it wraps.

export SIGNED_PIPELINE_SECRET='my secret'

buildkite-signed-pipeline upload

Verifying a pipeline signature

In a global environment hook, you can include the following to ensure that all jobs that are handed to an agent contain the correct signatures:

export SIGNED_PIPELINE_SECRET='my secret'

if ! buildkite-signed-pipeline verify ; then
  echo "Step verification failed"
  exit 1
fi

This step will fail if the provided signatures aren't in the environment. The tool allows buildkite-signed-pipeline upload to be executed without a signature, this allows the initial upload step to be entered into the Buildkite UI.

Managing signing secrets

Simple secret

Per the examples above, the secret for signing and verification can be provided via an environment variable or command line flag.

AWS SM

This tool also has first-class support for AWS Secrets Manager (AWS SM). A secret id or ARN can be provided, the secret value will then be fetched from AWS SM to be used for signing and verification.

export SIGNED_PIPELINE_AWS_SM_SECRET_ID='arn:aws:secretsmanager:ap-southeast-2:12345:secret:my-signed-pipeline-secret-42a5qP'

buildkite-signed-pipeline upload

Future versions of the tool will add support for secret versioning.

How it works

When the tool receives a pipeline for upload, it follows these steps:

Iterates through each step of a JSON pipeline
Extracts the command or commands block
Trims whitespace on resulting command
Calculates HMAC(SHA256, command + BUILDKITE_BUILD_ID + canonicalised(BUILDKITE_PLUGINS), shared-secret)
Add STEP_SIGNATURE={hash} to the step environment block
Pipes the modified JSON pipeline to buildkite-agent pipeline upload

When the tool is verifying a pipeline:

Calculates HMAC(SHA256, BUILDKITE_COMMAND + BUILDKITE_BUILD_ID + canonicalised(BUILDKITE_PLUGINS), shared-secret)
Compare result with STEP_SIGNATURE
Fail if they don't match

Note that in the current version of this tool the secret is symmetric -- it's the same for signing/verifying.

Attack scenarios

For reference, this tool considers at least the following attack scenarios:

A malicious user gains access to the Buildkite UI (buildkite.com), and updates pipeline settings (adds/modifies a command or plugin)

Commands cannot be signed without knowing the signing secret ✅

Buildkite is compromised and arbitrary jobs (commands) are sent to all known agents

Only pipelines from your repositories will be signed ✅
Make sure your agents check BUILDKITE_REPO to ensure only known repositories are cloned ⚠️

The command (`BUILDKITE_COMMAND`) for a job is changed by a man-in-the-middle between Buildkite.com and your agents

The job signature validation will fail as it will not match the command from the uploaded pipeline ✅

A plugin parameter is changed (e.g. `docker` `image`) by a man-in-the-middle to a poisoned Docker image

The job signature validation will fail as it will not match the plugin from the uploaded pipeline ✅

A malicious plugin is added to a known "allow listed command" by a man-in-the-middle

This tool requires that jobs with plugins are signed, regardless of the allowed command ✅

A trusted plugin is compromised and malicious code is injected

If this vector is a concern, you can pin plugins to commit hashes vs versions ⚠️

A malicious user gains access to your build agents

This tool will not help in this scenario ❌

The signing secret is leaked/stolen

With the right signing secret, any command/plugins combination can be signed (and thus trusted by your agents) ❌

A malicious user gains access to your allow-listed git repositories (e.g. on GitHub)

This tool will not help in this scenario ❌

Development

This is using Golang's 1.11 modules.

export GO111MODULE=on
go run .

buildkite-signed-pipeline's People

Contributors

Stargazers

Watchers

Forkers

redbubble cawilson

buildkite-signed-pipeline's Issues

Add first class support for AWS Secrets Manager

E.g. the tool could have support for AWS Secrets Manager by providing an ARN to the secret.

export SIGNED_PIPELINE_AWS_SM_SECRET_ARN="aws:..."
buildkite-signed-pipeline upload

Add documentation for usage within the Docker/Docker Compose plugin

The Docker and Docker Compose plugins mount the buildkite-agent binary into the Docker container. When uploading pipelines (e.g. dynamic pipelines), this tool will also need to be mounted

Signer does not understand new Buildkite `group` and `matrix` steps

The pipeline-signer does not understand the new Buildkite group step. Commands in these steps are not signed, and therefore fail during the "checking signature" phase (with the Signature mismatch. error).

This is because the signer expects "steps" to be a child of the root of the pipeline yaml. Instead, any command in any part of the pipeline file should be signed.

Buildkite refs:

I assume the code around this area should change:

https://github.com/buildkite/buildkite-signed-pipeline/blob/master/cmd/buildkite-signed-pipeline/signer.go#L50

Also to note - I have not tested this but I assume the new Build Matrix matrix property will not work, for the same reasons.

Prevent tampering of BUILDKITE_PLUGINS

Buildkite plugins can reference things beyond the command (e.g. a Docker image)

Add support for whitelisting plugins

This could include plugin versions, and would replace https://github.com/buildkite/buildkite-plugin-whitelister

@lox has an implementation for working with plugins (for the agent): https://github.com/buildkite/agent/pull/762/files#diff-e46069f38e288ad2a2206cab78dcbde7

Crashes when pipelines use list syntax for environment variables

Pipelines can list env vars as lists, similar to docker-compose.

steps:
  - command: true
    env:
      - MY_VAR=value

This causes the pipeline signing to fail because it expects a map:

https://github.com/seek-oss/buildkite-signed-pipeline/blob/b598adbed132529869ed0f1167a0c524f697c8bd/cmd/buildkite-signed-pipeline/signer.go#L134-L140


panic: reflect: call of reflect.Value.MapKeys on slice Value
--
  |  
  | goroutine 1 [running]:
  | reflect.flag.mustBe(0x97, 0x15)
  | /usr/local/Cellar/go/1.11.1/libexec/src/reflect/value.go:207 +0xb4
  | reflect.Value.MapKeys(0x850340, 0xc00049c220, 0x97, 0x3, 0xc0002fa528, 0x1)
  | /usr/local/Cellar/go/1.11.1/libexec/src/reflect/value.go:1133 +0x49
  | main.SharedSecretSigner.signStep(0xc00046c080, 0x20, 0x0, 0x0, 0x876e60, 0xc00049c250, 0x194, 0x1, 0xc0001ae600, 0x0, ...)
  | /Users/zsims/projects/buildkite-signed-pipeline/signer.go:137 +0x50e
  | main.SharedSecretSigner.Sign(0xc00046c080, 0x20, 0x0, 0x0, 0x878960, 0xc0001ae0f0, 0x7f742a184000, 0x0, 0xc000066ba0, 0x0)
  | /Users/zsims/projects/buildkite-signed-pipeline/signer.go:57 +0x668
  | main.(*uploadCommand).run(0xc00000ae40, 0xc00016bc20, 0x859000, 0xc000035450)
  | /Users/zsims/projects/buildkite-signed-pipeline/main.go:100 +0xfa
  | main.(*uploadCommand).run-fm(0xc00016bc20, 0x409af3, 0x88d720)
  | /Users/zsims/projects/buildkite-signed-pipeline/main.go:38 +0x34
  | gopkg.in/alecthomas/kingpin%2ev2.(*actionMixin).applyActions(0xc0000c20d8, 0xc00016bc20, 0x0, 0x0)
  | /Users/zsims/go/pkg/mod/gopkg.in/alecthomas/[email protected]/actions.go:28 +0x6d
  | gopkg.in/alecthomas/kingpin%2ev2.(*Application).applyActions(0xc0000c4690, 0xc00016bc20, 0x0, 0x0)
  | /Users/zsims/go/pkg/mod/gopkg.in/alecthomas/[email protected]/app.go:557 +0xdf
  | gopkg.in/alecthomas/kingpin%2ev2.(*Application).execute(0xc0000c4690, 0xc00016bc20, 0xc000035440, 0x1, 0x1, 0x0, 0x0, 0xc00016e960, 0x0)
  | /Users/zsims/go/pkg/mod/gopkg.in/alecthomas/[email protected]/app.go:390 +0x8f
  | gopkg.in/alecthomas/kingpin%2ev2.(*Application).Parse(0xc0000c4690, 0xc00001c090, 0x3, 0x3, 0x1, 0xc00000c1e8, 0x0, 0x1)
  | /Users/zsims/go/pkg/mod/gopkg.in/alecthomas/[email protected]/app.go:222 +0x1fa
  | main.main()
  | /Users/zsims/projects/buildkite-signed-pipeline/main.go:80 +0xa6a

Add support for multiple secrets (versions)

E.g. secrets could have an ID, allowing rotation of secrets while also supporting builds that are in progress. Alternatively, this could be an array of secrets that are used for verification (until success)

Handle empty plugin and env blocks

env or plugins can be blank, which will cause this tool to blow up:

steps:
  - command: echo hi
    plugins:

2019/04/09 04:18:02 Unknown plugin type

Plugins are re-ordered causing signature mis-matches

In some cases, the plugins are re-ordered.

The "canonical" plugins reported by this tool:

[{"github.com/buildkite-plugins/docker-buildkite-plugin#v2.0.0":{"volumes":["./:/usr/local/src/example"],"environment":["STAGE","SYNC_PATH"]}}]

But on the step BUILDKITE_PLUGINS is

[{"github.com/buildkite-plugins/docker-buildkite-plugin#v2.0.0":{"environment":["STAGE","SYNC_PATH"],"volumes":["./:/usr/local/src/example"]}}]

Infer the AWS SM region from the ARN

Per https://github.com/seek-oss/buildkite-signed-pipeline/blob/e2bedb1643ac2c6aa22619512c0d24db3ad8d92e/main.go#L192 this assumes the "current" region is the same region of the secret. Even if the ARN is used, the --region still needs to be set on the SDK in order to read it from AWS SM.

Support "no command eval" style allow listing of (unsigned) commands

This is in addition to #14. The Buildkite UI still needs to be used to specify an "initial" command. This command will currently be unsigned.

This feature adds a "no command eval" style approach that any file in the repository can be executed unsigned provided the rules from #14 are true (e.g. no plugins are referenced, and the step doesn't have a signature)

Use a constant time comparison for checking the signatures

This is to mitigate against timing attacks, e.g. https://verboselogging.com/2012/08/20/a-timing-attack-in-action

Support plugins without arguments

This currently errors when there's a plugin being referenced that doesn't take arguments, e.g.:

steps:
  - command: 'echo wow'
    plugins:
      - seek-oss/docker-ecr-cache#v0.0.1:
      - docker#v2.0.0:

Results in

2018/10/23 09:42:22 $ buildkite-agent pipeline upload --dry-run

2018-10-23 09:42:22 INFO   Searching for pipeline config...

2018-10-23 09:42:22 INFO   Found config file ".buildkite/pipeline.yml"

panic: interface conversion: interface {} is string, not map[string]interface {}



goroutine 1 [running]:

main.SharedSecretSigner.extractPlugins(0xc000406d50, 0x6, 0x0, 0x0, 0x852460, 0xc000410020, 0x878f40, 0xc0003d8230, 0x94, 0x10)

	/Users/zsims/projects/buildkite-signed-pipeline/signer.go:154 +0x5b5

main.SharedSecretSigner.signStep(0xc000406d50, 0x6, 0x0, 0x0, 0x878f40, 0xc000188000, 0x194, 0xc0003d81d0, 0x94, 0xc00011bac8, ...)

	/Users/zsims/projects/buildkite-signed-pipeline/signer.go:104 +0x984

main.SharedSecretSigner.Sign(0xc000406d50, 0x6, 0x0, 0x0, 0x87aa40, 0xc000074810, 0xc000000300, 0xc00011bc40, 0x81065c, 0xc00008c238)

	/Users/zsims/projects/buildkite-signed-pipeline/signer.go:57 +0x668

main.(*uploadCommand).run(0xc00007ce40, 0xc000185c20, 0x85b0e0, 0xc000073400)

	/Users/zsims/projects/buildkite-signed-pipeline/main.go:98 +0xf4

main.(*uploadCommand).run-fm(0xc000185c20, 0x409af3, 0x88f900)

	/Users/zsims/projects/buildkite-signed-pipeline/main.go:41 +0x34

gopkg.in/alecthomas/kingpin%2ev2.(*actionMixin).applyActions(0xc00018a0d8, 0xc000185c20, 0x0, 0x0)

	/Users/zsims/go/pkg/mod/gopkg.in/alecthomas/[email protected]/actions.go:28 +0x6d

gopkg.in/alecthomas/kingpin%2ev2.(*Application).applyActions(0xc0001ac690, 0xc000185c20, 0x0, 0x0)

	/Users/zsims/go/pkg/mod/gopkg.in/alecthomas/[email protected]/app.go:557 +0xdf

gopkg.in/alecthomas/kingpin%2ev2.(*Application).execute(0xc0001ac690, 0xc000185c20, 0xc0000733f0, 0x1, 0x1, 0x0, 0x0, 0xc0001a6960, 0x0)

	/Users/zsims/go/pkg/mod/gopkg.in/alecthomas/[email protected]/app.go:390 +0x8f

gopkg.in/alecthomas/kingpin%2ev2.(*Application).Parse(0xc0001ac690, 0xc00007c030, 0x1, 0x1, 0x1, 0xc00008c1d0, 0x0, 0x1)

	/Users/zsims/go/pkg/mod/gopkg.in/alecthomas/[email protected]/app.go:222 +0x1fa

main.main()

	/Users/zsims/projects/buildkite-signed-pipeline/main.go:79 +0x914

�[31m🚨 Error: The command exited with status 2�[0m