Code Monkey home page Code Monkey logo

github-release-resource's Introduction

GitHub Releases Resource

Fetches and creates versioned GitHub resources.

If you're seeing rate limits affecting you then please add a token to the source configuration. This will increase your number of allowed requests.

Source Configuration

  • owner: Required. The GitHub user or organization name for the repository that the releases are in.

  • repository: Required. The repository name that contains the releases.

  • access_token: Optional. Used for accessing a release in a private-repo during an in and pushing a release to a repo during an out. The access token you create is only required to have the repo or public_repo scope.

  • github_api_url: Optional. If you use a non-public GitHub deployment then you can set your API URL here.

  • github_v4_api_url: Optional. If you use a non-public GitHub deployment then you can set your API URL for graphql calls here.

  • github_uploads_url: Optional. Some GitHub instances have a separate URL for uploading. If github_api_url is set, this value defaults to the same value, but if you have your own endpoint, this field will override it.

  • insecure: Optional. Default false. When set to true, concourse will allow insecure connection to your github API.

  • release: Optional. Default true. When set to true, check detects final releases and put publishes final releases (as opposed to pre-releases). If false, check will ignore final releases, and put will publish pre-releases if pre_release is set to true

  • pre_release: Optional. Default false. When set to true, check detects pre-releases, and put will produce pre-releases (if release is also set to false). If false, only non-prerelease releases will be detected and published.

    note: if both release and pre_release are set to true, put produces final releases and check detects both pre-releases and releases. In order to produce pre-releases, you must set pre_release to true and release to false.
    note: if both release and pre_release are set to false, put will still produce final releases.
    note: releases must have semver compliant tags to be detected.

  • drafts: Optional. Default false. When set to true, put produces drafts and check only detects drafts. If false, only non-draft releases will be detected and published. Note that releases must have semver compliant tags to be detected, even if they're drafts.

  • semver_constraint: Optional. If set, constrain the returned semver tags according to a semver constraint, e.g. "~1.2.x", ">= 1.2 < 3.0.0 || >= 4.2.3". Follows the rules outlined in https://github.com/Masterminds/semver#checking-version-constraints.

  • tag_filter: Optional. If set, override default tag filter regular expression of v?([^v].*). If the filter includes a capture group, the capture group is used as the release version; otherwise, the entire matching substring is used as the version.

  • order_by: Optional. One of [version, time]. Default version. Selects whether to order releases by version (as extracted by tag_filter) or by time. See check behavior described below for details.

Example

- name: gh-release
  type: github-release
  source:
    owner: concourse
    repository: concourse
    access_token: abcdef1234567890
- get: gh-release
- put: gh-release
  params:
    name: path/to/name/file
    tag: path/to/tag/file
    body: path/to/body/file
    globs:
    - paths/to/files/to/upload-*.tgz
    generate_release_notes: true

To get a specific version of a release:

- get: gh-release
  version: { tag: 'v0.0.1' }

To set a custom tag filter:

- name: gh-release
  type: github-release
  source:
    owner: concourse
    repository: concourse
    tag_filter: "version-(.*)"

Behavior

check: Check for released versions.

Lists releases, sorted either by their version or time, depending on the order_by source option.

When sorting by version, the version is extracted from the git tag using the tag_filter source option. Versions are compared using semver semantics if possible.

When sorting by time and a release is published, it uses the publication time, otherwise it uses the creation time.

The returned list contains an object of the following format for each release (with timestamp in the RFC3339 format):

{
    "id": "12345",
    "tag": "v1.2.3",
    "timestamp": "2006-01-02T15:04:05.999999999Z"
}

When check is given such an object as the version parameter, it returns releases from the specified version or time on. Otherwise it returns the release with the latest version or time.

in: Fetch assets from a release.

Fetches artifacts from the requested release.

Also creates the following files:

  • tag containing the git tag name of the release being fetched.
  • version containing the version determined by the git tag of the release being fetched.
  • body containing the body text of the release.
  • timestamp containing the publish or creation timestamp for the release in RFC 3339 format.
  • commit_sha containing the commit SHA the tag is pointing to.
  • url containing the HTMLURL for the release being fetched.

Parameters

  • globs: Optional. A list of globs for files that will be downloaded from the release. If not specified, all assets will be fetched.

  • include_source_tarball: Optional. Enables downloading of the source artifact tarball for the release as source.tar.gz. Defaults to false.

  • include_source_zip: Optional. Enables downloading of the source artifact zip for the release as source.zip. Defaults to false.

out: Publish a release.

Given a name specified in name, a body specified in body, and the tag to use specified in tag, this creates a release on GitHub then uploads the files matching the patterns in globs to the release.

Parameters

  • name: Required. A path to a file containing the name of the release.

  • tag: Required. A path to a file containing the name of the Git tag to use for the release.

  • tag_prefix: Optional. If specified, the tag read from the file will be prepended with this string. This is useful for adding v in front of version numbers.

  • commitish: Optional. A path to a file containing the commitish (SHA, tag, branch name) that the release should be associated with.

  • body: Optional. A path to a file containing the body text of the release.

  • globs: Optional. A list of globs for files that will be uploaded alongside the created release.

  • generate_release_notes: Optional. Causes GitHub to autogenerate the release notes when creating a new release, based on the commits since the last release. If body is specified, the body will be pre-pended to the automatically generated notes. Has no effect when updating an existing release. Defaults to false.

Development

Prerequisites

  • golang is required - version 1.15.x is tested; earlier versions may also work.
  • docker is required - version 17.06.x is tested; earlier versions may also work.

Running the tests

The tests have been embedded with the Dockerfile; ensuring that the testing environment is consistent across any docker enabled platform. When the docker image builds, the test are run inside the docker container, on failure they will stop the build.

Run the tests with the following command:

docker build -t github-release-resource --target tests --build-arg base_image=concourse/resource-types-base-image-static:latest .

Contributing

Please make all pull requests to the master branch and ensure tests pass locally.

github-release-resource's People

Contributors

ahelal avatar aoldershaw avatar areller avatar chenbh avatar coro avatar defsprite avatar drnic avatar evashort avatar jleben avatar jmcarp avatar jmelchio avatar jtarchie avatar kaleo211 avatar kirillbilchenko avatar krobertson avatar mariash avatar markstokan avatar matt-royal avatar mdb avatar oppegard avatar robdimsdale avatar shinji62 avatar snordhausen avatar taylorsilva avatar thephilderbeast avatar vito avatar xoebus avatar xtremerui avatar youssb avatar zachgersh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

github-release-resource's Issues

get doesn't show what was added via a previous put

When I put to an existing github-release in the pipeline to append a file, the next job which also does a get against the same resource doesn’t see the recently-added file. Indeed when I hijack the next job and examine the resource, I see the usual body, tag, and version for the github-release resource but not the recently-added file. The Github UI does show that file present for the release indicating that it was successfully added.

The concourse documentation for put indicates:

When the put succeeds, the produced version of the resource will be immediately fetched via an implicit get step. This is so that later steps in your plan can use the artifact that was produced.

This expected behavior works in a different resource type like the regular git resource, but does not appear to work in the github-release resource. Is this expected behavior?

Filter checked versions by regex or prefix.

Add feature to filter versions by regex or at least prefix matching. For example if I wanted to fetch the latest 4.x version of a release.

- name: gh-release
  type: github-release
  source:
    version_match: '4\..*'

or

- name: gh-release
  type: github-release
  source:
    version_prefix: '4.'

The ultimate goal is that all versions returned by check will be major version 4.

support for self signed certificates

is there a way to skip ssl cert validation? we have a proxy that has a self signed cert on it and this will cause the download to fail with the error

error running command: Get https://api.github.com/repos/enaml-ops/omg-cli/releases: x509: certificate signed by unknown authority

can an option be added to ignore this?

Cannot start Docker container intermittently with 'cf create-service'

The issue is intermittent. Typically, a retry works. Any thoughts and how do we troubleshoot?

212442468@cf-sb-adm001:~/workspace/applications/rabbitmq-cloudfoundry-samples/spring$ cf cs logstash free delete-rabbitmq-logstash
Creating service instance delete-rabbitmq-logstash in org CSOps / space riak as admin...
FAILED
Server error, status code: 502, error code: 10001, message: Service broker error: #<Exceptions::BackendError: Cannot start Docker container `cf-6619f8df-b422-4769-933b-de4ce63095cc'>

Feature request: Use github repository deployment keys to authenticate

Github provides Oauth2 access tokens on a per-user basis to authenticate against their API, which you use.

If I understand correctly, the token scope granularity is only on a public / private repo basis, not on a per-repo basis. That is to say, a token with the repo scope will give the token permissions on all of the private repositories a user can access, and the public_repo scope does the same for all public repos the user can access.

If a team wants to use this feature, then an individual team member's credentials must be used in the concourse pipeline with this resource. This means that 1) other team members may have access to repos they should not and 2) when that team member rolls off, the credential must be changed.

Github also provides access on a per-repository basis with deployment SSH keys. It would be nice if github-release-resource could use these keys to authenticate, so that teams with multiple members do not need to put an individual team member's credentials in their concourse pipeline.

resource fails to find releases

When the github-release resource attempts to check a repository for releases which appears to be created as a tag rather than a release, the resource fails to find any releases

The nginx repository releases are an example of one I've found so far https://github.com/nginx/nginx/releases
Is there a way to detect if a new release is added here using this resource, or is there a better resource to use?

Create release as a pre-release, upload all files, then change to published release

The moment that a release is created (non pre-release) then any other concourse pipeline that watches it can trigger - except that the release might not yet be finished. If the finished release includes several large files, then it is possible for downstream pipelines to trigger too early.

Perhaps if we create the release as a pre-release, then upload files, then switch to a final release we will have happy downstream pipelines?

pipeline does not find github resource

We have a new pipeline that is dependent on a github release. The pipeline is hung on 'waiting for a suitable set of input versions`

Typing a check resource command using the --from option seems to solve it. It would be nice concourse would do this for us.

Publish a release not working

When i try to publish a release to GitHub i get the following error
error running command: GET https://api.github.com/repos//DashVid-Uploader/releases: 404 Not Found []

My resource is setup as follows

- name: gh-release
  type: github-release
  source:
    owner: JassoftLtd
    repository: DashVid-Uploader
    access_token: {{gh_access_token}}

It appears as though the owner isn't being put in the URL?
as https://api.github.com/repos/JassoftLtd/DashVid-Uploader/releases works for me

Non-idempotent releases with same tag should fail

We have the following snippet in our pipeline

  plan:
  - get: gpos_src
    passed:
    - gpos_centos5_release
  - get: bin_gpos_centos5_release
    passed:
    - gpos_centos5_release
    trigger: true
  - task: gpos_publish_tag
    file: gpos_src/concourse/publish_tag.yml
  - put: gpos_github_release
    params:
      name: gpos_github_release_stage/tag.txt
      tag: gpos_github_release_stage/tag.txt
      commitish: gpos_github_release_stage/commit.txt
      globs:
        - gpos_github_release_stage/bin_gpos_centos5_release.tar.gz

Currently if the job is triggered twice with:

  1. different commits in gpos_src
  2. different tar balls from bin_gpos_centos5_release
  3. same content in the tag.txt file

Then we observe the following behavior:

  1. Both put will succeed
  2. The value of the tag stays with the commit from the first put
  3. The files attached are from the second put

Is it reasonable that we should expect the second put to flat out fail?

github release creation fails - invalid Message

On every attempt to push to make a github release this error pops up:

POST https://api.github.com/repos/drahnr/oregano/releases: 422 Validation Failed [{Resource:Release Field:target_commitish Code:invalid Message:}]

https://ci.spearow.io/teams/main/pipelines/oregano/jobs/make-release/builds/14

https://screenshots.firefox.com/BazokjFlfCeCyiHN/ci.spearow.io

fly -t spearow hijack -j oregano/make-release                                                                                      1: build #14, step: gh-release, type: put
2: build #14, step: tag2file, type: task
choose a container: 2
[root@5c9cc5a8-543d-4228-563c-c1b6d9905eac 64658831]# cat 
deb-release/      git-tag-resource/ release-info/     rpm-release/      source-release/   
[root@5c9cc5a8-543d-4228-563c-c1b6d9905eac 64658831]# cat release-info/
cat: release-info/: Is a directory
[root@5c9cc5a8-543d-4228-563c-c1b6d9905eac 64658831]# cat release-info/
BODY       COMMITISH  NAME       TAG        
[root@5c9cc5a8-543d-4228-563c-c1b6d9905eac 64658831]# cat release-info/
BODY       COMMITISH  NAME       TAG        
[root@5c9cc5a8-543d-4228-563c-c1b6d9905eac 64658831]# cat release-info/*
Yet another release
9c14571
oregano
0.84.21

Local check

#git log -n1

commit 9c14571b0cf6b693cb8df5bb85439431aba05a34 (HEAD -> master, tag: v0.84.21, origin/master, origin/HEAD)
Author: Bernhard Schuster <[email protected]>
Date:   Sat Dec 16 10:11:21 2017 +0100

    version/bump: v0.84.21

Relevant concourse.yml

resources:
- name: git-tag-resource
  type: git
  source:
    tag_filter: "v[0-9]*"
    branch: master
    uri: https://github.com/drahnr/oregano.git

- name: gh-release
  type: github-release
  source:
    owner: drahnr
    repository: oregano
    access_token: {{drahnr-github-token}}

-jobs:
  - name: make-release
    build_logs_to_retain: 5
    public: true
    plan:
    - aggregate:
      - get: git-tag-resource
        passed: [source-release,build-pkg-rpm-release, build-pkg-deb-release]
        trigger: true
      - get: oregano-test-fedora
      - get: source-release
        passed: [source-release]
      - get: rpm-release
        passed: [build-pkg-rpm-release]
      - get: deb-release
        passed: [build-pkg-deb-release]
    - task: tag2file
      timeout: 5m
      image: oregano-test-fedora
      config:
        platform: linux
        inputs:
        - name: git-tag-resource
        - name: source-release
        - name: deb-release
        - name: rpm-release
        outputs:
        - name: release-info
        run:
          path: sh
          args:
          - -exc
          - |
            git log -n 1 --format="%h" HEAD > ../release-info/COMMITISH
            ./waf version
            cat VERSION > ../release-info/TAG
            echo "oregano" > ../release-info/NAME
            echo "Yet another release" > ../release-info/BODY
          dir: git-tag-resource
    - put: gh-release
      params:
        tag_prefix: v
        name: release-info/NAME
        tag: release-info/TAG
        body: release-info/BODY
        commitish: release-info/COMMITISH
        globs:
        - source-release/oregano-*.tar.xz
        - rpm-release/oregano-*.rpm
        - deb-release/oregano_*.deb

Option to fetch solely release number

I'd like to use this resource to fetch the latest release number available for an external project, but download absolutely none of the assets (I'm using the Docker file provided on Docker Hub instead). However globs downloads everything by default.

Failed to create github final release with 404 error

ci/pipeline.yml

- name: github-final-release
  type: github-release
  source:
    github_api_url: https://github.build.ge.com/api/v3/
    user: industry-cloud-cf
    repository: cf-sensu-client-release
    access_token: {{github-access-token}}

github-final-release error reported

error running command: GET https://github.build.ge.com/api/v3/repos/industry-cloud-cf/cf-sensu-client-release/releases: 404 Not Found []

resource script '/opt/resource/out [/tmp/build/put]' failed: exit status 1

Failing to put release

I'm getting the following error when trying to put a release in a private repo.
creating release ReleaseName error running command: POST https://api.github.com/repos/my-org/my-repo/releases: 404 Not Found []

prior to getting this error, I was getting:
error running command: GET https://api.github.com/repos/my-org/my-repo.git/releases: 404 Not Found []
so I know the get is now working, and it's something specific tot he POST.

My resource config is as follows (admittedly doctored to protect the innocent)

- name: gh-release
  type: github-release
  source:
    owner: my-org
    repository: my-repo
    access_token: {{access-token}}

and the put looks like this (again doctored):

    - put: gh-release
      params:
        name: package/name
        tag: version/version
        tag_prefix: package/tag-prefix
        commitish: package/commitish
        globs:
        - package/*.tar.gz

I know the access token works for the GET, and I've used it with curl successfully.

Any ideas what I might be doing wrong?

Failing with return code 413 when uploading blob

Error: error running command: POST https://uploads.github.com/repos/pivotal-cf/concourse-olm/releases/6990340/assets?name=olm-0.26.0.tar.gz: 413 []

Seems to fail with Request entity too large code. But when uploading the file manually through github.com UI it seems to publish it fine.

blob is of size 440.0M.
screen shot 2017-07-10 at 4 30 24 pm

Suggestion: draft option to prevent releases from being published automatically

The current behavior of this is resource is a little surprising. If a draft exists with the same name of the release, the release draft is updated appropriately (including the draft tag), but no real tagging occurs. However, if no draft exists, the release is published immediately. It would be awesome if there were a draft option that would prevent the release from being published, even if it doesn't exist as a draft already.

Not downloading the newest files

I have a pipeline similar to ci.concourse.ci.
After creating a RC version it builds the source, uploads to github release, builds helm charts and merges with github release binaries, uploads both charts and binaries -> deploys.

When I just rebuild the charts and upload them to the github release the deploy step always deploys the old charts. When I download the charts manually from github to check, they are actually the correct, new charts.

Is there any cache for resources in concourse? Maybe something is wrong with my pipeline. Not sure.

    serial: true
    plan:
      - aggregate:
          - get: source
            passed: [rc]
            trigger: true
          - get: version
            passed: [rc]
            trigger: true
      - aggregate:
          - task: build
 - name: build-charts
    serial: true
    plan:
      - aggregate:
          - get: source
            passed: [build]
            trigger: true
          - get: charts
            trigger: true
          - get: version
            passed: [build]
            trigger: true
          - get: rc
            passed: [build]
            trigger: true
      - task: build-charts
      - put: github-release
...
  - name: deploy-rc
    serial: true
    interruptible: true
    plan:
      - aggregate:
          - get: source
            passed: [build-charts]
          - get: version
            passed: [build-charts]
            trigger: true
          - get: config
            passed: [config-smoke-test]
            trigger: true
          - get: rc
            passed: [build-charts]
            trigger: true
            params:
              globs:
                - '*-chart.tgz'
    - deploy

[in] store metadata about release in files

During in I'd like to know the git tag & calculated semver (e.g. remove any prefix v from git tag) to use for creating downstream assets. Perhaps store in <resource name>/gittag & <resource name>/version respectively?

Knowing the tag would allow the git resource to fetch the corresponding tag of the repo (e.g. to access the specific spiff templates from cf-release)

`in` should create a file, `release_url`, which is the URL of the release

In our php-notifications pipeline, we use a number of different mechanisms to consume upstream dependencies. One of these is github-release-resource, another is new_version_resource.

new_version_resource can supply the URL of a page that was scraped for new versions. This means that our PHP Module Checker job, which posts a story to tracker, can supply URLs for dependencies we're updating.

It would be nice to get a release_url file from github-release-resource so that we could do the same for dependencies tracked this way.

[Idea] Body with Tag placeholder

Hi I would like to make a quick proposal about adding a tag for example #tag placeholder into the body option .

For example tag is (0.4.0) so I can declare the body filename like

body: dir/filename_#tag resolve to dir/filename_0.4.0
body: dir/file#tagname resolve to dir/file0.4.0name

I am willing to made a PR if that interest someone.

Error when hitting GitHub rate limiter

When you hit the rate limiter from GitHub it causes this resource to fail instead of producing a helpful error message.

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x47d137]

goroutine 1 [running]:
github.com/concourse/github-release-resource.(*GitHubClient).ListReleaseAssets(0xc82006d500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /tmp/build/dfd4986f-bce1-4ec6-4d09-aeb063f7f16d/gopath/src/github.com/concourse/github-release-resource/github.go:144 +0x97
github.com/concourse/github-release-resource.(*InCommand).Run(0xc8200c5ee0, 0x7ffd652c1f27, 0xe, 0xc820090c80, 0x16, 0xc82006b9e0, 0x5, 0x0, 0x0, 0x0, ...)
    /tmp/build/dfd4986f-bce1-4ec6-4d09-aeb063f7f16d/gopath/src/github.com/concourse/github-release-resource/in_command.go:65 +0x721
main.main()
    /tmp/build/dfd4986f-bce1-4ec6-4d09-aeb063f7f16d/gopath/src/github.com/concourse/github-release-resource/cmd/in/in.go:27 +0x308

goroutine 34 [IO wait]:
net.runtime_pollWait(0x7f2b8e0d1608, 0x72, 0xc82006a0b0)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82008b870, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82008b870, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82008b810, 0xc8200bb800, 0x800, 0x800, 0x0, 0x7f2b8e0cc028, 0xc82006a0b0)
    /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8200e4010, 0xc8200bb800, 0x800, 0x800, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc8200e0360, 0x7f2b8e08c080, 0xc8200e4010, 0x5, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:455 +0xcc
crypto/tls.(*Conn).readRecord(0xc8200f4000, 0x89f717, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:540 +0x2d1
crypto/tls.(*Conn).Read(0xc8200f4000, 0xc8200fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:901 +0x167
net/http.noteEOFReader.Read(0x7f2b8e094b48, 0xc8200f4000, 0xc8200c6108, 0xc8200fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc820419700, 0xc8200fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc82039af00)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc82039af00, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8200c60b0)
    /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 35 [select]:
net/http.(*persistConn).writeLoop(0xc8200c60b0)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

failing to discover previous versions

- name: homebrew
  type: github-release
  check_every: 12h
  source:
    owner: Homebrew
    repository: brew

This only reports the latest version available which is preventing me from going back to a previous version (the latest version has broken me).

e.g. the above resource currently sees version 1.7.2 but I need 1.7.1

silently failing to download and stage source release file

with concourse 4.0.0:

I have a pipeline which grabs a github release archive:

- name: homebrew
  type: github-release
  check_every: 12h
  source:
    owner: Homebrew
    repository: brew

and

    - get: homebrew
      params:
        include_source_tarball: true

The pipeline was working fine, then I put my computer to sleep with the stack 'stopped', then resumed my pc and woke the stack back up. The pipeline ran again as expected, and showed a valid version for the homebrew resource.

However, when it gets to the step in the pipeline where it uses the file, there's no .tar.gz file to be found!

+ cd /opt/concourse/local/worker/volumes/live/7df5d723-4cb4-4239-7b73-d3c6857ebe6c/volume/homebrew
+ ls -la
total 0
drwxr-xr-x  2 eedwards  staff   64 Aug 16 10:41 .
drwxr-xr-x  6 root      staff  192 Aug 17 09:40 ..

So for whatever reason the check container is reporting success, and concourse is acting like it pulled and staged the resource correctly, but the binary file is missing!

Also, a fly check-resource succeeds, but the file is still missing!

Get latest from github-api-url/latest instead of github-api-url/0

Instead of specifying the latest release and having to update it every time, the resource could pick up the latest.

Currently a new github-release resource tries to query:

https://api.github.com/repos/cloudfoundry/cf-deployment-concourse-tasks/releases/0

This fails with a 404. It should check /latest when no release version is specified in the resource:

https://api.github.com/repos/cloudfoundry/cf-deployment-concourse-tasks/releases/latest

source.tar.gz does not match Github release tar.gz

The release tarball from the Github release page does differ from the source.tar.gz provided by the Github Release Resource.

If the tarball is downloaded from the Github release page, the main directory inside of the tarball has the project name and the current version.
The source.tar.gz on the other side got the project name and part of a hash.

For example this tarball https://github.com/cloudfoundry/cli/archive/v6.32.0.tar.gz has the directory name cli-6.32.0 and the one from the github resource with the same version and in theory content has cloudfoundry-cli-829c87e.

It would be great if either the github-release-resource source.tar.gz would have the same content as the official tarball from the Github page or there is an option get this behavior.

Especially for building packages and similar it would be helpful to have the version in the directory name, so no extra intelligence is needed to determine the first parts of some commit hash.

Create blank files on IN when no release exists yet?

I recently ran into an issue where I am trying to run my pipeline through for the very first time, and because there is no release in github yet, a job that is trying to determine the next version number can never run.

Being able to specify default values for version, tag, & body to store in the files in the case that no releases are found at all, would help prevent this issue.

`pre_release: true` is ineffective by itself

Running Concourse 2.6.0.

What We Expect

If we have a github-release-resource with pre_release: true, we expect all put operations to create a pre-release on Github.

We expect that no other configuration is necessary.

What We Observe

When we perform a put, we just get a standard release, not a pre-release.

However, when we add release: false as well as pre_release: true, we observe that put gives us a pre-release.

Feature Request: Support for filtering versions

We were trying to auto bump some of the software components that we use in our project. This seemed like the perfect resource to do that. But, we did not want to automatically bump major versions. It seems like the resource does not support filtering versions. I would ideally like to specify version filter like < 4.0.0 to specify that I do not want any 4.x.y version, but all releases to 3.x.y should trigger the pipeline.

Get fails for drafted release if the associated tag is a draft tag

We get this panic after creating a release that is merged into a draft, during the get. This happens because this line will return an empty release when one does not exist, as the tag provided to GetReleaseByTag does not actually exist.

Steps to reproduce:

  1. Setup a draft of a release in Github
  2. Create a concourse job that publishes a release with the same name as the draft release
  3. Confirm that the tag that concourse will create does not already exist in the repo
  4. Start job

Strangely, this seemed to be working on our previous (0.59.0) concourse deployment.

Latest version fetching failing

Hello!

We think we are configuring our concourse pipeline correctly to pick up our latest release, but we are getting an error.

Resource:

- name: bbr-binary-release
  type: github-release
  source:
      owner: cloudfoundry-incubator
      repository: bosh-backup-and-restore

Used in plan as

  - get: bbr-binary-release
    trigger: true

According to the readme:

If the version is not specified, the latest version is chosen using semver semantics.

We want it to fetch the latest (non-draft) release, and our release tags look like they are semver-compatible

But when it runs in concourse, we get the following error

error running command: GET https://api.github.com/repos/cloudfoundry-incubator/bosh-backup-and-restore/releases/0: 404 Not Found []

If we configure a version in the plan it works, so:

  - get: bbr-binary-release
    version: { tag: "v1.1.0" }

works.

cc: @flangewad

Publishing of draft releases is broken

After upgrade to v3.9.0 (and v3.9.1) publishing of draft release is failing.
Actually, draft release is creating, but subsequent get operation fails with following message:

error running command: GET https://api.github.com/repos/myuser/myrepo/git/refs/tags/mydrafttag: 404 Not Found []

The same pipeline works well on v3.8.0. Looks like it was probably broken by #64

Resource ignores tag and always pulls the latest

We are unable to pin to a specific release using this resource.

---
resources:
- name: concourse-release
  type: github-release
  source:
    user: concourse
    repository: concourse
    access_token: YOUR-GITHUB-ACCESS-TOKEN-GOES-HERE
jobs:
- name: test-concourse-release
  plan:
  - do:
    - get: concourse-release
      version: { tag: 'v2.4.0' }
    - task: test-release
      config:
        platform: linux
        image_resource:
          type: docker-image
          source:
            repository: ubuntu
        inputs:
        - name: concourse-release
        run:
          path: sh
          args:
          - -exc
          - |
            find .
            cat ./concourse-release/tag
            cat ./concourse-release/version

The output of this task is:

waiting for a suitable set of input versions
concourse-release - pinned version {"tag":"v2.4.0"} is not available

Even though the tag exists: https://github.com/concourse/concourse/releases/tag/v2.4.0

This behavior exists for other repos and tags as well.

Auto-unpack source tarball/zip

I'm under the impression the Concourse s3 resource can unpack source tarballs on download. I have the same use case for download gh-release source code-- I would like to download the release source code & have it automatically unpacked in the same directory.

Currently I'm implementing the logic in a task, but it would save me quite a bit of code if the source code contents could be untarred/unzipped automatically.

      - task: unpack-tarball
        config:
          platform: linux
          image_resource:
            type: docker-image
            source: {repository: czero/cflinuxfs2}
          inputs: [name: tarball]
          outputs: [name: untarred]
          run:
            path: bash
            args: [-c, tar -xvf tarball/*.tar.gz -C untarred --strip-components=1]

Thanks for your time & consideration

[question] should the resource create annotated tags instead of lightweight tags?

From the git documentation at git tag --help:

Annotated tags are meant for release while lightweight tags are meant for private or temporary object labels. For this reason, some git commands for naming objects (like git describe) will ignore lightweight tags by default.

How do you feel about switching to annotated tags then?

Failed to upload release to github with 401 bad credentials

Able to access the repo with security token, however failed to upload release to github.

curl https://{github-access-token}:@github.build.ge.com/api/v3/repos/industrial-cloud-cf/cf-sensu-client-release/releases
- name: github-final-release
  type: github-release
  source:
    github_api_url: https://github.build.ge.com/api/v3/
    user: industrial-cloud-cf
    repository: cf-sensu-client-release
    access_token: {{github-access-token}}

creating release cf-sensu-client-release v1.6.0
uploading /tmp/build/put/create-final-release/boshrelease/releases/cf-sensu-client/cf-sensu-client-1.6.0.tgz
error running command: POST https://uploads.github.com/repos/industrial-cloud-cf/cf-sensu-client-release/releases/547/assets?name=cf-sensu-client-1.6.0.tgz: 401 Bad credentials []

resource script '/opt/resource/out [/tmp/build/put]' failed: exit status 1

Nil pointer deference

Not sure what would have caused this, just manually triggered a build.

https://diego.ci.cf-app.com/pipelines/main/jobs/ketchup-deploy-msi/builds/527

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x47d137]

goroutine 1 [running]:
github.com/concourse/github-release-resource.(*GitHubClient).ListReleaseAssets(0xc820011590, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /tmp/build/dfd4986f-bce1-4ec6-4d09-aeb063f7f16d/gopath/src/github.com/concourse/github-release-resource/github.go:144 +0x97
github.com/concourse/github-release-resource.(*InCommand).Run(0xc8200b5ee0, 0x7ffcdec62f28, 0xe, 0xc82000ad00, 0x16, 0xc82000ace0, 0x15, 0x0, 0x0, 0xc820011440, ...)
    /tmp/build/dfd4986f-bce1-4ec6-4d09-aeb063f7f16d/gopath/src/github.com/concourse/github-release-resource/in_command.go:65 +0x721
main.main()
    /tmp/build/dfd4986f-bce1-4ec6-4d09-aeb063f7f16d/gopath/src/github.com/concourse/github-release-resource/cmd/in/in.go:27 +0x308

goroutine 11 [IO wait]:
net.runtime_pollWait(0x7f448c874730, 0x72, 0xc82000e160)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82000d950, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82000d950, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82000d8f0, 0xc8200a3000, 0x400, 0x400, 0x0, 0x7f448c86f050, 0xc82000e160)
    /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc82002c0f0, 0xc8200a3000, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc820011f50, 0x7f448c874b78, 0xc82002c0f0, 0x5, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:455 +0xcc
crypto/tls.(*Conn).readRecord(0xc8200c02c0, 0x89f717, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:540 +0x2d1
crypto/tls.(*Conn).Read(0xc8200c02c0, 0xc8200e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:901 +0x167
net/http.noteEOFReader.Read(0x7f448c87d410, 0xc8200c02c0, 0xc8200b6108, 0xc8200e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc8203f2760, 0xc8200e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc82033f7a0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc82033f7a0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8200b60b0)
    /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 12 [select]:
net/http.(*persistConn).writeLoop(0xc8200b60b0)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

Concourse pipeline to download the .pivotal from corporate git hub release

Hi Team,

From concourse pipeline, trying to download the .pivotal from corporate Git hub release, but while connecting to Git hub release, URL is appending with /repo/, due to that not able to connect to the Git hub release:
Configurations used to connect to Git hub release from concourse pipeline:

  • name: pcf-custom-tiles
    type: github-release
    source:
    user: OpenPlatform
    repository: xxx-tile
    paths: [xxxxxxxxxxxxxxxx.pivotal]
    github_api_url: {{GIT_URI}}
    access_token: xxxxxxxxxxxxxxxxxxxxxx

while connecting, it reforms the git hub release URL as below:
Get https://github.xxxx.com/repos/OpenPlatform/xxx-tile/releases
It appends the /repo/ to the URL, due to that, the connection is failing,
Can we over ride the Git Hub URL ?
Please suggest on this.

Thank you.
Raghavendra

cannot specify older version in `get`

Hi,

We have a use-case with the following repo:
https://github.com/PowerShell/Win32-OpenSSH/releases

Very old version of the release have timestamps, eg 5_15_2016. The resource in a new pipeline never picks up the latest versions because it thinks these timestamps are the newest ones.

It would be nice if we could specify a tag in the resource definition. Otherwise we are forced to run fly check-resource every time we stand up a new pipeline or a new version comes out, which breaks the promise of statelessness in concourse pipelines.

fail when download asset does not contain the redirect URL

We've had issues that the get of the resource caches an incorrect value for the assets.

For example:

$ echo '{"source":{"user":"pivotal-cf-experimental","repository":"concourse-windows-worker-release"}, "version": {"tag":"3.3.3"}}' | /opt/resource/in /tmp
$ cat /tmp/resource.tgz
{"url":"https://api.github.com/repos/pivotal-cf-experimental/concourse-windows-worker-release/releases/assets/4426005" ... }

We were expecting the release.tgz to have been a large tarball. It appears that the Github API for assets returns a location. When we don't receive a location, we return the contents of the API call, as seen above.

We need a better error message.

Body of draft release is always overridden

When the resource updates an existing release draft, any release notes in that draft are clobbered, even if no body is specified. I would expect the draft body to be left intact if no body is specified.

Unusual version filtering/ sorting

While using this resource with the BOSH releases, old versions are getting discovered as latest. Where the latest release is v263 the release being detected is stable-3363.24.

After doing some investigation it became increasingly unclear what the expected behaviour of this resource is with regards to release version filtering and sourcing. After looking through commit history I found: 7331e0a containing the note NOTE: There is also strict filtering on only allowing semver supported tags. (the note is also in the README Note that releases must have semver compliant tags to be detected.) using v?([^v].*) as a regex matcher. Based on that commit it appears as though it was not the intention to match tags like stable-3363.24.

In order to match only match semver compliant versions I swapped out the determineVersionFromTag with below:

func determineVersionFromTag(tag string) string {
  re := regexp.MustCompile(`^v?(\d+\.)?(\d+\.)?(\*|\d+)$`)
  matches := re.FindStringSubmatch(tag)
  if len(matches) > 0 {
    return strings.Join(matches[1:], "")
  }
  return ""
}

On doing this and running the tests I discovered a number of failing tests, one example is:

Expected
    <[]resource.Version | len:2, cap:2>: [
        {Tag: "0.4.0", ID: ""},
        {Tag: "v0.4.2", ID: ""},
    ]
to equal
    <[]resource.Version | len:6, cap:6>: [
        {Tag: "0.4.0", ID: ""},
        {Tag: "0.4.1-rc.9", ID: ""},
        {Tag: "v0.4.1-rc.10", ID: ""},
        {Tag: "v0.4.2-rc.1", ID: ""},
        {Tag: "v0.4.2", ID: ""},
        {Tag: "v0.4.3-rc.1", ID: ""},
    ]

This test appears to be matching versions that are not strictly semver versions but can easily be matched by swapping the above regex to ^v?(\d+\.)?(\d+\.)?(\*|\d+.*)$ or even using a slightly modified version of the original regex doing a less strict match ^v?([^v].*).

During my investigations I also found that the github library that is being used makes it relively easy to filter based on the latest published version using the RepositoryRelease.PublishedAt field, the sort function would look something like:

type byVersion []*github.RepositoryRelease

func (r byVersion) Len() int {
  return len(r)
}

func (r byVersion) Swap(i, j int) {
  r[i], r[j] = r[j], r[i]
}

func (r byVersion) Less(i, j int) bool {
  return r[i].PublishedAt.Before(r[j].PublishedAt.Add(0))
}

Some example filter/ sorted outputs:

Current:

v260.1
v260.2
v260.3
v260.4
v260.5
v260.6
v261
v261.1
v261.2
v261.3
v261.4
v262
v262.1
v262.2
v262.3
v263
stable-3312.9
stable-3312.12
stable-3312.15
stable-3312.17
stable-3363
stable-3363.1
stable-3363.9
stable-3363.10
stable-3363.14
stable-3363.15
stable-3363.19
stable-3363.20
stable-3363.22
stable-3363.24

Strict Semver:

v260.1
v260.2
v260.3
v260.4
v260.5
v260.6
v261
v261.1
v261.2
v261.3
v261.4
v262
v262.1
v262.2
v262.3
v263

Timestamp:

stable-3312.9       2016-12-15 07:43:18 +0000 UTC
v260.1              2016-12-20 03:16:26 +0000 UTC
stable-3312.12      2016-12-20 19:24:14 +0000 UTC
v260.2              2017-01-12 07:56:36 +0000 UTC
stable-3312.15      2017-01-13 03:00:18 +0000 UTC
v260.3              2017-01-14 00:16:44 +0000 UTC
v260.4              2017-01-21 02:16:34 +0000 UTC
v260.5              2017-01-24 00:00:34 +0000 UTC
stable-3312.17      2017-02-02 23:23:53 +0000 UTC
v260.6              2017-02-03 21:29:34 +0000 UTC
v261                2017-02-16 02:15:09 +0000 UTC
stable-3363         2017-02-16 02:36:29 +0000 UTC
v261.1              2017-02-17 18:50:41 +0000 UTC
stable-3363.1       2017-02-18 00:46:10 +0000 UTC
v261.2              2017-02-22 02:43:15 +0000 UTC
stable-3363.9       2017-02-23 02:46:25 +0000 UTC
v261.3              2017-03-06 18:39:33 +0000 UTC
stable-3363.10      2017-03-09 00:04:27 +0000 UTC
v261.4              2017-03-14 16:56:19 +0000 UTC
stable-3363.14      2017-03-31 00:41:58 +0000 UTC
stable-3363.15      2017-04-05 23:16:00 +0000 UTC
stable-3363.19      2017-04-17 22:53:28 +0000 UTC
stable-3363.20      2017-04-25 23:31:04 +0000 UTC
stable-3363.22      2017-05-15 17:43:00 +0000 UTC
stable-3363.24      2017-05-22 21:48:14 +0000 UTC
v262                2017-06-06 13:41:50 +0000 UTC
v262.1              2017-06-10 02:40:25 +0000 UTC
v262.2              2017-06-28 22:03:41 +0000 UTC
v262.3              2017-07-06 18:00:58 +0000 UTC
v263                2017-08-14 19:33:31 +0000 UTC

Happy to raise a pull request to impliment any of the above but some guidance around what the correct direction is would be extremely helpful.

  • How strict do we want to be with Semver versions?
  • Should semver version matching be a configuration option?
  • Should we provide options on sorting via configuration (PublishedAt or highest semver)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.