According to the json schema this is a deprecated format

It'd be nice if you guys could implement an net/http.Handler that automatically parses incoming webhooks into useful structs.

Something like what I've implemented here: https://godoc.org/github.com/omise/omise-go#WebhookHTTPHandler

The CreateBuild type does not allow setting pull request information. Would you accept a PR to add this functionality?

See my conversation with Buildkite support:

Describe the bug

Buildkite has a heretogenous definitions of steps. This means, steps can have very different formats. Take the wait steps for example, it has none of the keys the command steps have.

This breaks the way Go wants to work with pipeline steps as they are currently tied to a single type called Step. Ideally this should come from an interface (something like PipelineStep) that allows different types of step to implement their own decoding strategy.

To Reproduce

Given the below pipeline:

# pipeline.yaml
---

steps:
  - name: "Command 1"
    command: ls

  - wait: ~

  - name: "Command 2"
    command: ls

The below code fails to correctly Unmarshall the YAML to the Go structure:

pipeline := &buildkite.Pipeline{}

pipelineYAMLBytes, err := os.ReadFile("./pipeline.yaml")

if err != nil {
	panic(err)
}

err = yaml.Unmarshal(pipelineYAMLBytes, pipeline)

if err != nil {
	panic(err)
}

waitStepBytes, err := yaml.Marshal(pipeline.Steps[1])

if err != nil {
	panic(err)
}

fmt.Println(string(waitStepBytes))

The output of the wait step is an empty step, which makes it impossible to work with this data as you wouldn't be able to tell what the step is supposed to do. Note that any other step type other than the command step will have the same behaviour.

Expected behavior
A clear and concise description of what you expected to happen.

Additional context

This hinders the ability for the SDK to be used in various common usecases of BK.

If the BK team is interested, I'm happy to work alongside you to create a workable solution.

Thanks.

build.failing is a type of webhook event that can be sent, but it doesn't seem supported by this library

It seems that it's missing from this mapping https://github.com/buildkite/go-buildkite/blob/main/buildkite/webhook.go#L24, and also from parsing

We want to pick out the Author of a build since Creator is only correct if the author has a Buildkite user.

If you look at

The JSON event that Buildkite sends does contain the Author...

Here is a sample event that Buildkite sends copied (and censored) from the Builkite Webhook Notification page:

It has the Author field in there already and it will only match if the email in Author matches an existing user in Buildkite with the same email.

{
  "event": "build.finished",
  "build": {
    "id": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "graphql_id": "xxxxxxxxxxxxxxxxxxxxxxxxxx=",
    "number": 20125,
    "state": "passed",
    "blocked": false,
    "blocked_state": "",
    "message": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "commit": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "branch": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "tag": null,
    "source": "api",
    "author": {
      "name": "n.n",
      "email": "[email protected]"
    },
    "creator": {
      "id": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
      "graphql_id": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
      "name": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
      "email": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
      "avatar_url": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
      "created_at": "2019-12-16T14:11:42.986Z"
    },
    "created_at": "2021-03-23T13:21:52.185Z",
    "scheduled_at": "2021-03-23T13:21:52.109Z",
    "started_at": "2021-03-23T13:22:00.000Z",
    "finished_at": "2021-03-23T13:56:55.000Z",
    "meta_data": {
      "ci": "True",
      "change": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
      "message": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
      "fullbuild": "True",
      "is_tryjob": "true",
    },
    "pull_request": null,
    "rebuilt_from": null
  },
  "pipeline": {
    "id": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "graphql_id": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "url": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "web_url": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "name": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "description": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "slug": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "repository": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "cluster_id": null,
    "branch_configuration": "",
    "default_branch": "pioneer",
    "skip_queued_branch_builds": false,
    "skip_queued_branch_builds_filter": null,
    "cancel_running_branch_builds": false,
    "cancel_running_branch_builds_filter": null,
    "allow_rebuilds": true,
    "provider": {
      "id": "github",
      "settings": {
        "trigger_mode": "none",
        "build_pull_requests": true,
        "pull_request_branch_filter_enabled": false,
        "skip_builds_for_existing_commits": false,
        "skip_pull_request_builds_for_existing_commits": true,
        "build_pull_request_forks": false,
        "build_pull_request_ready_for_review": false,
        "prefix_pull_request_fork_branch_names": true,
        "build_tags": false,
        "publish_commit_status": false,
        "publish_commit_status_per_step": false,
        "separate_pull_request_statuses": false,
        "publish_blocked_as_pending": false,
        "filter_enabled": false,
        "repository": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
        "pull_request_branch_filter_configuration": ""
      },
      "webhook_url": "xxxxxxxxxxxxxxxxxxxxxxxxxx"
    },
    "builds_url": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "badge_url": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "created_by": {
      "id": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
      "graphql_id": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
      "name": "M M",
      "email": "[email protected]",
      "avatar_url": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
      "created_at": "2019-09-03T08:24:46.059Z"
    },
    "created_at": "2019-10-03T07:12:15.505Z",
    "archived_at": null,
    "env": null,
    "scheduled_builds_count": 0,
    "running_builds_count": 13,
    "scheduled_jobs_count": 3,
    "running_jobs_count": 28,
    "waiting_jobs_count": 7,
    "visibility": "private",
    "configuration": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "steps": [
      {
        "type": "script",
        "name": ":pipeline:",
        "command": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
        "artifact_paths": null,
        "branch_configuration": null,
        "env": {
        },
        "timeout_in_minutes": null,
        "agent_query_rules": [
          "queue=ue4-ci-enqueue"
        ],
        "concurrency": null,
        "parallelism": null
      }
    ]
  },
  "sender": {
    "id": "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    "name": "xxxxxxxxxxxxxxxxxxxxxxxxxx"
  }
}

Describe the bug

Hi team. I am seeing an issue with this method in the Golang SDK for Buildkite pipelines:

_, errBuildkite := buildkiteClient.Pipelines.AddWebhook(owner, repositoryName)

When I try to add a webhook to an existing pipeline, instead of receiving a 201 as expected (https://buildkite.com/docs/apis/rest-api/pipelines#add-a-webhook), I get a 404:

POST https://api.buildkite.com/v2/organizations/Wesfarmers-Digital/pipelines/myrepo/webhook: 404 Not Found

I confirmed that the repository exists in Github, that the pipeline exists in Buildkite too.
Out of 6+ tries, only 2 succeeded

To Reproduce
Steps to reproduce the behavior:

1. Create a github repository using this github golang cdk function: client.Repositories.CreateFromTemplate (github.com/google/go-github/v54/github)
2. Create a buildkite pipeline for this repository using this buildkite golang cdk function: buildkiteClient.Pipelines.Create
3. Create a webhook on the github repository through this buildkite golang cdk function: buildkiteClient.Pipelines.AddWebhook
4. See error

Expected behavior
I would expect to receive a 201 HTTP return code.

Additional context
I repeated the webhook creation step a few times after creating the repository and the pipeline.

What are the plans for team support - is it being looked at?
Edit: obvs, this is a side glance at graphql

If I understand correctly, it seems the docker-compose.yml used by go-buildkite configures the use of Go 1.13 in CI, despite that the go.mod cites Go 1.16. I would think it's preferable to consistently use the Go version cited in go.mod in CI too.

Hi Team, hopefully this is right place to ask, if not, I'd appreciate if you can direct me.

I'm the founder of cloudquery.io, a high performance open source ELT framework.

Our users are interested in a Buildkite plugin, but as we cannot maintain all the plugins ourselves, I was curious if this would be an interesting collaboration, where we would help implement an initial source plugin, and you will help maintain it (Similar to terraform provider but much simpler as it is just the Extract part).

This will give your users the ability to sync Buildkite APIs (jobs, config,...) to any of their datalakes/data-warehouses/databases easily using any of the growing list of CQ destination plugins.

Best,
Yevgeny

So suddenly this morning the go-buildkite library randomly stopped working. The issue appears as most fields in the agent list appearing as nil.

curl -s -H 'Authorization: Bearer <token>' "https://api.buildkite.com/v2/organizations/org/agents" - worked
curl -s -H 'Authorization: Bearer <token>' --user-agent 'go-buildkite/2.0.0' "https://api.buildkite.com/v2/organizations/org/agents" - was missing information

It looks like the server is sending a different response when the user agent is go-buildkite/2.0.0, not sure why that would be, but I assume you have your reasons.

That said I thought I was using v2.1.0 of the library!

Then I found this: https://github.com/buildkite/go-buildkite/blob/v2.1.0/buildkite/version.go

It looks like the version number wasn't updated in the latest release.

So, can we please get a new release of go-buildkite with a working user agent? The one in master seems fine ...

Hi BK team,

are there any simplified way to test code that relies on go-buildkite other than injecting a custom RoundTripper to the underlying client transport?

For the go-github package, which apparently inspired this repo, there's a unittesting library built by the community called https://github.com/migueleliasweb/go-github-mock. Is there anything like it so unittesting is less painful?

https://godoc.org/github.com/buildkite/go-buildkite/buildkite Usually that's the first thing someone is checks when finding a Go library so probably nice to add.

unblocked_at is a property on block steps. However, this field is not present on the job struct. When using the client, unblocked_at field is unattainable for block steps in API responses.

The job struct should contain the unblocked_at field and should be obtainable through API calls using this client.

These fields are missing in the Job struct, even though they exist in the Jobs API docs: https://buildkite.com/docs/apis/rest-api/jobs

Describe the bug
unblocked_at is a property on block steps. However, this field is not present on the job struct. When using the client, unblocked_at field is unattainable for block steps in API responses.

To Reproduce
Any APIs returning block step will not have the unblocked_at field.

Expected behavior
The job struct should contain the unblocked_at field and should be obtainable through API calls using this library.

Looks like the TokenConfig and NewBasicConfig have the DefaultTransport hardcoded. It would be nice if they accepted a user provided transport and optionally used that in case the user wants to configure TLS, timeouts, etc.

Hello! When will the next release being created? I've recently submitted a change and would like to get a new release so that we can start to use that field.

Thanks!

We want to use go-buildkite to programmatically get/set a Github Enterprise pipeline's setting called "trigger_mode" and set it to "none", which is equivalent to the "Disable Github Activity" option from BK UI.

But it appears go-buildkite SDK does not have this option for GithubEnpterpriseSettings but it exists for GithubSettings. And so we have to use a workaround in our code to update the pipeline setting with the API call.

I'm working on some template-tooling that (among other things) automates buildkite set-up when folks want to create a new service or stack. It doesn't do any of the remote execution work - only creating the .buildkite directory with a bunch of defaults; creating Cloudformation/Terraform stuff for the CI stacks; and things like that. It's then up to the project team to execute things remotely.

This package** is pretty great for dealing with the remote API but it feels a little bit coupled, in a sense.

Would it be a Bad Idea ™️ to separate the data models from the service implementation and auth?

Like, to build something using the remote API:

import (
  "github.com/buildkite/go-buildkite/buildkite"
  "github.com/buildkite/go-buildkite/buildkite/models"
)

But, to build something using only the data models (for file generation):

import (
  "github.com/buildkite/go-buildkite/buildkite/models"
)

That should make the current API stable:

client := buildkite.NewClient(config.Client())
projects, _, err := client.Projects.List(*org, nil)

I know there's always a back and forth on Go projects about flat or nested package spaces... I'm happy to do the refactoring, but don't want to do so if it clashes with the maintainers' preferences, as it's more a preference than anything else.

(not yet OSS - pending some company conversations)
*(It's awesome to see the go-buildkite project in the buildkite org, btw!)

If you pass nil as the last parameter to the UnblockJob function, the request receives a 500 Internal Server Error from Buildkite.

If you pass &buildkite.JobUnblockOptions{ Fields: map[string]string{} }, then go-buildkite generates the request query string Fields=map%5B%5D, which is invalid, but ignored by the API.

The expected behavior is that both nil and &buildkite.JobUnblockOptions{ Fields: map[string]string{} } work and generate a valid request URL.

Hey! Looks like version 2.2.0 was released 5 days ago; but the tag was not updated to reflect this (which most dep managers need to have the version instead of latest)

I think the solution is as simple as tagging master with v2.2.0

Pipeline struct is missing created_by

It has the line

config, err := buildkite.NewTokenConfig(*apiToken)

which is missing a second (bool) parameter to buildkite.NewTokenConfig(...).

In local development using Go 1.19, it appears to me that make fails with...

$ make
go test -timeout=3s -v ./...
# github.com/buildkite/go-buildkite/v3/buildkite
buildkite/pipelines.go:102:10: fmt.Errorf call has more than one error-wrapping directive %w
FAIL    github.com/buildkite/go-buildkite/v3/buildkite [build failed]
?       github.com/buildkite/go-buildkite/v3/examples/artifacts [no test files]
?       github.com/buildkite/go-buildkite/v3/examples/projects  [no test files]
FAIL
make: *** [test] Error 2

If I understand correctly, this is because:

It is invalid to include more than one %w verb

...as explained at https://pkg.go.dev/fmt#Errorf

I confess I'm not entirely sure why this evidently does not occur during CI, given that I don't believe Go 1.13's fmt.Errorf (used in CI) supports more than one %w, either. In any case, I'd think the error is probably worth addressing.

So many of the struct members that are returned by library calls are pointers. This presents a messy problem for third-party software using the library, because we have to check for nil pointers before accessing struct members. I see little advantage to using pointers here. It adds a lot of ugly checks. It's not a massive amount of data, so I think that it would be fine to just use regular data types, not pointers to data.

Thoughts?

Does the REST API handle pipelines with a slash in them?

I'm having trouble using the API to get a pipeline on my account with a slash in it. Pipelines without a slash in them work fine.

E.g. get on pipeline mtest works but get on pipeline lib/somefoo does not.

I've tried URL escaping the pipeline slug before sending it over and this too does not work.

Describe the bug
The build Author struct only contains a name and email. However, in a build.finished webhook, I see it also having a username field. It would be nice to have that in the struct as well to help when unmarshalling webhook events.

"author": {
  "name": "Matthew Dee",
  "username": "mdee",
  "email": "[redacted]"
},

Since uploading test results to Test Analytics is as simple as a curl, it would be slick if we could include it in go-buildkite :)
https://buildkite.com/docs/test-analytics/golang-collectors#main

When attempting to follow the usage, I'm getting the following issue:

> go get github.com/buildkite/go-buildkite/buildkite/v2
go get github.com/buildkite/go-buildkite/buildkite/v2: module github.com/buildkite/go-buildkite@upgrade found (v2.2.0+incompatible), but does not contain package github.com/buildkite/go-buildkite/buildkite/v2

When attempting to vendor with a manually updated go.mod, it looks like @latest hasn't been tagged with v2.3.1

> go mod vendor
go: finding module for package github.com/buildkite/go-buildkite/buildkite/v2
github.com/.../.../pkg/buildkite imports
	github.com/buildkite/go-buildkite/buildkite/v2: module github.com/buildkite/go-buildkite@latest found (v2.2.0+incompatible), but does not contain package github.com/buildkite/go-buildkite/buildkite/v2

Hello!

There's been many PRs merged since v3.0.1 (e.g. Webhook support). Would be nice to release a new version including all those changes 🙏🏼

When using the REST API to "list builds for a pipeline":

curl "https://api.buildkite.com/v2/organizations/{org.slug}/pipelines/{pipeline.slug}/builds"

the step_key field is part of the response for each job.

However, this value is not included with the response using the go-buildkite client (i.e. Builds.ListByPipeline()). Please expose this field. This makes it possible to query for the state of a particular step of a build. Thanks!

The package path is github.com/buildkite/go-buildkite/buildkite. That's buildkite three times. I propose at least removing one of them to github.com/buildkite/go-buildkite. Creating yet another Go package doesn't seem to be idiomatic Go from what I've seen elsewhere on Github.

When creating a build, the functions wants the following:
Create func(org string, pipeline string, b *CreateBuild) (*Build, *Response, error)

I have created an instance of the struct like the following and passed it to the function, but it doesn't seem to work.

buildkiteCreateBuildValues := buildkite.CreateBuild{ Commit: buildkiteCommit, Branch: buildkiteBranch, Message: buildkiteMessage, } createBuildResp, createBuildErr := buildkiteClient.Builds.Create(buildkiteOrganization, buildkitePipeline, *buildkiteCreateBuildValues)

Describe the bug
In order to write a test for a webhook handler, I would like to create sample webhook events, but cannot do so with a struct literal expression.

To Reproduce
Try to construct the following object:

&buildkite.JobFinishedEvent{Pipeline: &buildkite.Pipeline{}}

This gives a compiler error. However,

event := &buildkite.JobFinishedEvent{}
event.Pipeline = &buildkite.Pipeline{}

Expected behavior
It would be nice if the above pattern, or something similar, could be used to construct test objects.

Additional context

This happens because the type is

type JobFinishedEvent struct {
	jobEvent
}

and the embedded unexported field means we can't use a struct literal to create it, which is slightly annoying when writing tests.

In Golang, propagating context through the stack is very important:

1. Let you retain a certain metadata
2. Let you handle graceful shutdown logic
3. Let you implement cancel/timeout

Current library has not support for making http request with a context.

We could introduce RequestOptionFunc to help handle this (see https://github.com/xanzy/go-gitlab/blob/7d79a18484610e0d7b178392f020c4a8dfb419c6/applications.go#L75-L91 for reference)

// RequestOptionFunc can be passed to all API requests to customize the API request.
type RequestOptionFunc func(*retryablehttp.Request) error[Sander van Harmelen, 2 years ago: • Introduce client options](https://sourcegraph.com/github.com/xanzy/go-gitlab/-/commit/443faa191281c0e6dd2d6df9c29c2d1caa515d4d)

// WithContext runs the request with the provided context
func WithContext(ctx context.Context) RequestOptionFunc {
	return func(req *retryablehttp.Request) error {
		*req = *req.WithContext(ctx)
		return nil
	}
}

The modify existing interfaces to

func (*obj) ListObj(lo ListObjOptions, options ...RequestOptionFunc) ([]*Obj, *Response, error) {}

This would help it so that if parent application decided to shutdown half way through, we can cancel the in-flight request early (before the server even response)

https://github.com/buildkite/go-buildkite/blob/master/buildkite/pipelines.go#L79

It's not clear if name is supposed to be used in place of label, but if a Step struct is instantiated and then marshaled to YAML in order to pipe it to buildkite agent pipeline upload there is no way to specify the label field with this schema.