Code Monkey home page Code Monkey logo

cogito's Introduction

cogito

Cogito (COncourse GIT status resOurce) is a Concourse resource to update the GitHub commit status during a build. The name is a humble homage to RenΓ© Descartes.

It can also send a message to a chat system (currently supported: Google Chat). This allows to reduce the verbosity of a Concourse pipeline and especially to reduce the number of resource containers in a Concourse deployment, thus reducing load. Chat and GitHub commit status update can be used independently (see examples below).

Written in Go, it has the following characteristics:

  • As lightweight as possible (Docker Alpine image).
  • Extensive test suite.
  • Autodiscovery of configuration parameters.
  • No assumptions on the git repository (for example, doesn't assume that the default branch is main or that branch main even exists).
  • Supports Concourse instanced pipelines.
  • Helpful error messages when something goes wrong with the GitHub API.
  • Retryable GitHub HTTP API requests due to rate limiting or transient server errors.
  • Configurable logging for the three steps (check, in, out) to help troubleshooting.

Contributing and Development

This document explains how to use this resource. See CONTRIBUTING for how to build the Docker image, develop, test and contribute to this resource.

Please, before opening a PR, open a ticket to discuss your use case. This allows to better understand the why of a new feature and not to waste your time (and ours) developing a feature that for some reason doesn't fit well with the spirit of the project or could be implemented differently. This is in the spirit of Talk, then code.

We care about code quality, readability and tests, so please follow the current style and provide adequate test coverage. In case of doubts about how to tackle testing something, feel free to ask.

Semver, releases and Docker images

This project follows Semantic Versioning and has a CHANGELOG.

NOTE Following semver, no backwards compatibility is guaranteed as long as the major version is 0.

Releases are tagged in the git repository with the semver format vMAJOR.MINOR.PATCH (note the v prefix). The corresponding Docker image has tag MAJOR.MINOR.PATCH and is available from DockerHub.

Which Docker tag to use?

  • The latest tag always points to the latest release, not to the tip of master, so it is quite stable.
  • Alternatively, you can pin the resource to a specific release tag MAJOR.MINOR.PATCH.

Examples

Only GitHub commit status

See also pipelines/cogito.yml for a bigger example and for how to use YAML anchors to reduce as much as possible YAML verbosity.

resource_types:
- name: cogito
  type: registry-image
  check_every: 24h
  source:
    repository: pix4d/cogito

resources:
- name: gh-status
  type: cogito
  check_every: never
  source:
    owner: ((github-owner))
    repo: ((your-repo-name))
    access_token: ((github-PAT))

- name: the-repo
  type: git
  source:
    uri: https://github.com/((github-owner))/((your-repo-name))
    branch: ((branch))

jobs:
  - name: autocat
    on_success:
      put: gh-status
      inputs: [the-repo]
      params: {state: success}
    on_failure:
      put: gh-status
      inputs: [the-repo]
      params: {state: failure}
    on_error:
      put: gh-status
      inputs: [the-repo]
      params: {state: error}
    on_abort:
      put: gh-status
      inputs: [repo.git]
      params: {state: abort}
    plan:
      - get: the-repo
        trigger: true
      - put: gh-status
        inputs: [the-repo]
        params: {state: pending}
      - task: maybe-fail
        config:
          platform: linux
          image_resource:
            type: registry-image
            source: { repository: alpine }
          run:
            path: /bin/sh
            args:
              - -c
              - |
                set -o errexit
                echo "Hello world!"

GitHub commit status plus chat notification

The absolute minimum is adding key gchat_webhook to the source configuration of the previous example:

- name: gh-status
  type: cogito
  check_every: never
  source:
    owner: ((github-owner))
    repo: ((your-repo-name))
    access_token: ((github-PAT))
    gchat_webhook: ((gchat_webhook))

Chat notification only

The absolute minimum is setting keys gchat_webhook and sinks in the source configuration:

- name: chat-notify
  type: cogito
  check_every: never
  source:
    sinks:
      - gchat
    gchat_webhook: ((gchat_webhook))

Effects

Build states mapping

There is only a partial matching between Concourse and GitHub Commit status API, so we map as good as we can according to this table:

Concourse
State
Color and meaning GitHub
Commit status API
Chat notification
running 🟑 - running pending pending
success 🟒 - task exited 0 success success
failure πŸ”΄ - task exited non 0 failure failure
error 🟠 - any other error besides failure or abort (pipeline configuration error, network error, timeout, ...) error error
abort 🟀 - human-initiated abort error abort

The colors are taken from the Concourse UI and are replicated to the chat message.

Effects on GitHub

With reference to the GitHub Commit status API, the POST parameters (state, target_url, description, context) are set by Cogito and rendered by GitHub as follows:

Screenshot of GitHub UI

Effects on Google Chat

  • Create a Gchat space per pipeline or per group of related pipelines.
  • At space creation time (cannot be changed afterwards) select threaded messages.
  • In the upper left, click on the drop-down menu with the name of the space.
  • Click on Manage webhooks.
  • Create a webhook and securely store it in your Concourse secret management system.

Screenshot of Google Chat UI

Source Configuration

GitHub commit status only

Required keys

  • owner
    The GitHub user or organization.

  • repo
    The GitHub repository name.

  • access_token
    The OAuth access token.
    See also: section GitHub OAuth token.

Optional keys

  • context_prefix
    The prefix for the GitHub Commit status API "context" (see section Effects on GitHub). If present, the context will be set as context_prefix/job_name.
    Default: empty.
    See also: the optional context in the put step.

  • log_level:
    The log level (one of debug, info, warn, error).
    Default: info, recommended for normal operation.

  • github_hostname:
    GitHub hostname. This allows to post commit statuses to repositories hosted by GitHub Enterprise (GHE) instances. For example: github.mycompany.org will be expanded by cogito to https://github.mycompany.org/api/v3
    Default: github.com

  • log_url. DEPRECATED, no-op, will be removed
    A Google Hangout Chat webhook. Useful to obtain logging for the check step for Concourse < v7.x

GitHub commit status plus chat notifications

Required keys

Optional keys

  • The optional keys for Only GitHub commit status

  • chat_notify_on_states
    The build states that will cause a chat notification. One or more of abort, error, failure, pending, success.
    Default: [abort, error, failure].
    See also: section Build states mapping.

  • chat_append_summary
    One of: true, false. If true, append the default build summary to the custom put.params.chat_message and/or put.params.chat_message_file.
    Default: true.
    See also: the default build summary in Effects on Google Chat.

Chat notification only

Required keys

  • sinks
    The sinks list for chat notification. Acceptable values: [gchat].

  • gchat_webhook
    URL of a Google Chat webhook. A notification will be sent to the associated chat space.
    See also: chat_notify_on_states and section Effects on Google Chat.

Optional keys

  • chat_notify_on_states
    The build states that will cause a chat notification. One or more of abort, error, failure, pending, success.
    Default: [abort, error, failure].
    See also: section Build states mapping.

  • chat_append_summary
    One of: true, false. If true, append the default build summary to the custom put.params.chat_message and/or put.params.chat_message_file.
    Default: true.
    See also: the default build summary in Effects on Google Chat.

Suggestions

We suggest to set a long interval for check_interval, for example 24 hours, as shown in the example above. This helps to reduce the number of check containers in a busy Concourse deployment and, for this resource, has no adverse effects.

The check step

No-op. Will always return the same version, dummy.

The get step

No-op.

The put step

Sets the GitHub commit status for a given commit, following the GitHub Commit status API. The same commit can have multiple statuses, differentiated by parameter context.

If the source block has the optional key gchat_webhook, then it will also send a message to the configured chat space, based on the state parameter.

Required params

  • state
    The state to set. One of error, failure, pending, success, abort.
    See also: the mapping explained in section Effects.

Optional params for GitHub commit status

  • context
    The value of the non-prefix part of the GitHub Commit status API "context"
    Default: the job name.
    See also: Effects on GitHub, source.context_prefix.

Optional params for chat

  • sinks
    If present, overrides source.sinks. This allows to run step put for example to send to chat only. Default: source.sinks.

  • gchat_webhook
    If present, overrides source.gchat_webhook. This allows to use the same Cogito resource for multiple chat spaces.
    Default: source.gchat_webhook.

  • chat_message
    Custom chat message; overrides the build summary. Its presence is enough for the chat message to be sent, overriding source.chat_notify_on_states.
    Default: empty.

  • chat_message_file
    Path to file containing a custom chat message; overrides the build summary. Appended to chat_message. Its presence is enough for the chat message to be sent, overriding source.chat_notify_on_states.
    Default: empty.

  • chat_append_summary
    Overrides source.chat_append_summary.
    Default: source.chat_append_summary.

Note on the put inputs

If using only GitHub commit status (no chat), the put step requires only one "put inputs". For example:

on_success:
  put: gh-status
  inputs: [the-repo] # name of the git resource corresponding to the GitHub repo to be updated.
  params: {state: success}

If using both GitHub Commit status and the chat_message_file parameter, the put step requires only two "put inputs". For example:

on_success:
  put: gh-status
  # the-repo: git resource; the-message-dir: "output" of a previous task
  inputs: [the-repo, the-message-dir]
  params:
    state: success
    chat_message_file: the-message-dir/msg.txt

If using send to chat only and the chat_message_file parameter, the put step requires only one "put inputs". For example:

on_success:
  put: chat-only
  # the-message-dir: "output" of a previous task
  inputs: [the-message-dir]
  params:
    state: success
    chat_message_file: the-message-dir/msg.txt

If using send to chat only without chat_message_file parameter, inputs may be left empty.

on_success:
  put: gh-status
  inputs: []
  params:
    state: success
    chat_message: "This is the custom chat message."

The reasons of this strictness is to help you have an efficient pipeline, since if the "put inputs" list is not set explicitly, then Concourse will stream all inputs used by the job to this resource, which can have a big performance impact. From the "put inputs" documentation:

inputs: [string]

Optional. If specified, only the listed artifacts will be provided to the container. If not specified, all artifacts will be provided.

GitHub OAuth token

Follow the instructions at GitHub personal access token to create a personal access token.

Give to it the absolute minimum permissions to get the job done. This resource only needs the repo:status scope, as explained at GitHub Commit status API.

NOTE: The token is security-sensitive. Treat it as you would treat a password. Do not encode it in the pipeline YAML and do not store it in a YAML file. Use one of the Concourse-supported credentials managers, see Concourse credential managers.

See also the section Integration tests for how to securely store the token to run the end-to-end tests.

Caveat: GitHub rate limiting

From GitHub REST API:

Rate limiting

For API requests using Basic Authentication or OAuth, you can make up to 5000 requests per hour. All OAuth applications authorized by a user share the same quota of 5000 requests per hour when they authenticate with different tokens owned by the same user.

For unauthenticated requests, the rate limit allows for up to 60 requests per hour. Unauthenticated requests are associated with the originating IP address, and not the user making requests.

GitHub resets the limit once per hour (no sliding window). If rate limited, cogito will wait up to 15 minutes for the limit to clear, or fail immediately if it would have to wait more. The error message in the output of the put step will mention the cause.

License

This code is licensed according to the MIT license (see file LICENSE).

cogito's People

Contributors

aledegano avatar aliculpix4d avatar esysc avatar iamoric avatar lsjostro avatar marco-m avatar marco-m-pix4d avatar muzaffar-omer avatar odormond avatar tgolsson avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cogito's Issues

What json data should be passed onto STDIN when testing /opt/resource/out?

Apologies in advance if this is not the right place to ask questions, but I know no other options.

Within the cogito image, I'm trying to run the /opt/resource/out command by passing on a json data manually through STDIN for PUT. What is the exact format of the data that is required? I keep encountering some argument error from the json data. The task file looks like so:

---
platform: linux
image_resource:
  type: registry-image
  source:
    repository: ((redacted))/cogito-gh-status-resource
    tag: ((redacted))
    username: ((redacted))
    password: ((redacted))
inputs:
  - name: git-source
run:
  path: sh
  args:
    - -exc
    - |
      /opt/resource/out . < git-source/instana-tasks-resource/ci/request-data/request-data-populated.json

Support custom user configuration in integration tests

Hello,

As I was going through CONTRIBUTING.md to run the integration tests, and I modified the tasks.yml to input my own values from gopass into task test:env -- task test:all. I was wondering if it would be possible to open a PR to merge these changes.

Ex:

+++ b/Taskfile.yml
@@ -48,13 +48,16 @@ tasks:
     cmds:
       - '{{ .CLI_ARGS }}'
     env: &test-env
-      COGITO_TEST_COMMIT_SHA: '{{default "32e4b4f91bb8de500f6a7aa2011f93c3f322381c" .COGITO_TEST_COMMIT_SHA}}'
+      COGITO_TEST_COMMIT_SHA: 
+        sh: 'echo -n {{default "$(gopass show cogito/test_commit_sha)" .COGITO_TEST_COMMIT_SHA}}'
       COGITO_TEST_OAUTH_TOKEN:
-        sh: 'echo {{default "$(gopass show cogito/test_oauth_token)" .COGITO_TEST_OAUTH_TOKEN}}'
-      COGITO_TEST_REPO_NAME: '{{default "cogito-test-read-write" .COGITO_TEST_REPO_NAME}}'
-      COGITO_TEST_REPO_OWNER: '{{default "pix4d" .COGITO_TEST_REPO_OWNER}}'
+        sh: 'echo -n {{default "$(gopass show cogito/test_oauth_token)" .COGITO_TEST_OAUTH_TOKEN}}'
+      COGITO_TEST_REPO_NAME: 
+        sh: 'echo -n {{default "$(gopass show cogito/test_repo_name)" .COGITO_TEST_REPO_NAME}}'
+      COGITO_TEST_REPO_OWNER: 
+        sh: 'echo -n {{default "$(gopass show cogito/test_repo_owner)" .COGITO_TEST_REPO_OWNER}}'
       COGITO_TEST_GCHAT_HOOK:
-        sh: 'echo {{default "$(gopass show cogito/test_gchat_webhook)" .COGITO_TEST_GCHAT_HOOK}}'
+        sh: 'echo -n {{default "$(gopass show cogito/test_gchat_webhook)" .COGITO_TEST_GCHAT_HOOK}}'

--- a/github/commitstatus_test.go
+++ b/github/commitstatus_test.go
@@ -168,34 +168,34 @@ func TestGitHubStatusFailureIntegration(t *testing.T) {
                {
                        name:  "bad token: Unauthorized",
                        token: "bad-token",
-                       wantErr: `failed to add state "success" for commit 32e4b4f: 401 Unauthorized
+                       wantErr: fmt.Sprintf(`failed to add state "success" for commit %s: 401 Unauthorized
 Body: {"message":"Bad credentials","documentation_url":"https://docs.github.com/rest"}
 Hint: Either wrong credentials or PAT expired (check your email for expiration notice)
-Action: POST https://api.github.com/repos/pix4d/cogito-test-read-write/statuses/32e4b4f91bb8de500f6a7aa2011f93c3f322381c
-OAuth: X-Accepted-Oauth-Scopes: [], X-Oauth-Scopes: []`,
+Action: POST https://api.github.com/repos/%s/%s/statuses/%s
+OAuth: X-Accepted-Oauth-Scopes: [], X-Oauth-Scopes: []`, cfg.SHA[0:7], cfg.Owner, cfg.Repo, cfg.SHA),```

(If this was already possible to do, but I ran the commands incorrectly, please let me know, thanks!)

report commit author and message

Currently we send to the chat sink:

2023-07-24 09:04:37 UTC
pipeline foo-bar
job oo-bar-local-tests/1301
state πŸ”΄ failure
commit 123456 (repo: example/repo)

It would be nice to send also the commit author and message (no matter the state):

2023-07-24 09:04:37 UTC
pipeline foo-bar
job oo-bar-local-tests/1301
state πŸ”΄ failure
commit 123456 (repo: example/repo)

Same for the logging.

What we have:

[INFO]  cogito: This is the Cogito GitHub status resource. Tag: 0.10.1, commit: 05c92db, build date: 2023-05-31
[INFO]  cogito.put.ghCommitStatus: GitHub HTTP request: attempt=1 max=3
[INFO]  cogito.put.ghCommitStatus: commit status posted successfully: state=pending git-ref=123456

What we could have

[INFO]  cogito: This is the Cogito GitHub status resource. Tag: 0.10.1, commit: 05c92db, build date: 2023-05-31
[INFO]  cogito.put.ghCommitStatus: Processing commit git-ref=123456 author=johnny.stecchino comment="foo: add bar"
[INFO]  cogito.put.ghCommitStatus: GitHub HTTP request: attempt=1 max=3
[INFO]  cogito.put.ghCommitStatus: commit status posted successfully: state=pending git-ref=123456

When retrying an HTTP request to GitHub, Cogito does not print why

With the default log level, in case of retry, Cogito prints:

[INFO]  cogito.put.ghCommitStatus: GitHub HTTP request: attempt=1 max=3
[INFO]  cogito.put.ghCommitStatus: Sleeping for: time=4m47s

At line 2, the user can infer that the attempt at line 1 failed, but the reason (the HTTP status code) is not printed.
Instead, we should print the reason, because it would make easier to troubleshoot if needed.

FEATURE REQUEST: Dry-run flag

Dear Cogito maintainers,
Any chance you'd add a dry run configuration to your pipeline? The idea there is if a pipeline fails rather than alerting as it would do in the staging env, the pipeline step might no-op and alert the user that no request had gone out. We'd love such a feature for our staging concourse environment. I'd PR if you're open to it, or accept suggested implementations if there's something I'm not seeing in your readme.
Thanks!
Eli

Add pipeline name as prefix to context

I'm trying out concourse and setting up for a monorepo/multi-project workflow, and it's easy to end up having test/validate/build jobs in multiple pipelines for the same repository, etc. I'm wondering what your stance is on prefixing the pipeline name to the context, if the information is available? As an example, instead of just test for the test job it'd be for example publish-docker-image/test if this test is part of the publishing pipeline.

cogito: error: put: internal error: unexpected: negative sleep time: 0s

Just observed in our deployment:

[INFO]  cogito: This is the Cogito GitHub status resource. Tag: 0.10.1, commit: 05c92db, build date: 2023-05-31
[INFO]  cogito.put.ghCommitStatus: GitHub HTTP request: attempt=1 max=3
[INFO]  cogito.put.ghCommitStatus: Sleeping for: time=4m47s
[INFO]  cogito.put.ghCommitStatus: GitHub HTTP request: attempt=2 max=3
cogito: error: put: internal error: unexpected: negative sleep time: 0s

See also #123.

Build links for instanced pipelines don't work

We use Instanced Pipelines, and the build links we get for them in GitHub don't appear to work. Here is an example of a link from cogito:

https://concourse/teams/teamname/pipelines/pipeline_name/jobs/job_name/builds/5?vars=%7B%22DEPLOYMENTID%22%3A%222%22%2C%22DESC%22%3A%22Daily+tests+for+Preprod+2%22%2C%22TITLE%22%3A%22PREPROD+TESTS%22%7D

an example of a working link for the above build

https://concourse/teams/teamname/pipelines/pipeline_name/jobs/job_name/builds/5?vars.DEPLOYMENTID=%222%22&vars.DESC=%22Daily%20tests%20for%20Preprod%202%22&vars.TITLE=%22PREPROD%20TESTS%22

retry GitHub HTTP request when rate limited: does not sleep when it should

We detect the situation correctly, we calculate the correct time to sleep, we log that we will sleep but actually we do NOT sleep, thus we consume the 3 attempts one after the other and finally fail:

[INFO]  cogito: This is the Cogito GitHub status resource. Tag: 0.10.2, commit: d8221ad, build date: 2023-07-28
[INFO]  cogito.put.gChat: not sending to chat: reason="feature not enabled"
[INFO]  cogito.put.ghCommitStatus: GitHub HTTP request: attempt=1 max=3
[INFO]  cogito.put.ghCommitStatus: Sleeping for: duration=7m42s reason="rate limited, should retry"
[INFO]  cogito.put.ghCommitStatus: GitHub HTTP request: attempt=2 max=3
[INFO]  cogito.put.ghCommitStatus: Sleeping for: duration=7m42s reason="rate limited, should retry"
[INFO]  cogito.put.ghCommitStatus: GitHub HTTP request: attempt=3 max=3
[INFO]  cogito.put.ghCommitStatus: Sleeping for: duration=7m42s reason="rate limited, should retry"

cogito: error: put: failed to add state "pending" for commit XXX: 403 Forbidden
Body: {"message":"API rate limit exceeded for user ID YYY.","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}
Hint: Rate limited but the wait time to reset would be longer than 15m0s (MaxSleepTime)

This has been introduced by #128

😿

Support for Enterprise Github?

Does this line mean that for Enterprise Github (not github.com) will now work?

left := []string{"github.com", owner, repo}

fetching pix4d/cogito@sha256:cef4adfe17e9ee5f7c16a4094e435d88857f663f080507dcb6b61e7390b52852
29291e31a76a [==========================================] 2.7MiB/2.7MiB
ff9cc595764e [======================================] 274.7KiB/274.7KiB
60eff1c2ce14 [==============================================] 116b/116b
507e2309724c [==========================================] 5.9MiB/5.9MiB
This is the Cogito GitHub status resource. Tag: 0.6.0, commit: 5d34381a18, build date: 2021-08-26
resource source configuration and git repository are incompatible.
Git remote: "https://notgithub.com/xxx/xxx.git"
Resource config: host: github.com, owner: "xxx", repo: "xxx". wrong git remote

Provide a way to override BUILD_JOB_NAME === context on resources.

Right now the resource requires that you put all tasks that affect the same job status together in the same job within a pipeline because of the choice to make github "context" === BUILD_JOB_NAME. For reference, I believe this is where we set it. https://github.com/Pix4D/cogito/blob/master/resource/resource.go#L213-L215

I'd like to propose allowing users to pass in an optional parameter "context" that overrides the job name as the CONTEXT arg if not equal to "". Any objections or "gotchas" you can think of? Any reason you'd rather not have this property be configurable?

I'm happy to PR this if you think the feature has a chance.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.