Downloading from Internal Repo using Token about gradle-download-task HOT 10 CLOSED

Looks good to me. Thanks for the feedback.

As for the tips: If you like, you may replace the call to curl with new URL(...).openConnection() and use setRequestProperty() to set the authentication header (see here for an example in Groovy: https://stackoverflow.com/questions/25692515/groovy-built-in-rest-http-client)

Can we close the ticket now?

from gradle-download-task.

The HTTP request and response look correct. Either the repository/path does not exist or your credentials are incorrect.

I don't know if you're using your personal credentials, but you may try to use a GitHub access token instead. This usually helps, for example if you've enabled 2FA.

from gradle-download-task.

The unredacted version of the file /foo/bar/blob/master/model.tflite is accessible via firefox (after single sign on 2FA), so I know the path exists. Also these line:

    username System.getenv("GITHUB_USER")
    password System.getenv("GITHUB_TOKEN")

are making use of GitHub access tokens. The GITHUB_TOKEN here is an environmental variable that stores such a token. The token has been verified to not be expired, and verified to work with the following command:

curl -v -H "Authorization: token $GITHUB_TOKEN" -o model.tflite -L https://github.com/foo/bar/blob/master/model.tflite

By "work" above, I mean that it successfully downloads model.tflite using the same GitHub access token used by gradle.

from gradle-download-task.

A few things that might be of interest from the curl output:

* Connected to github.com (20.27.177.113) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]

...

* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=github.com
*  start date: Feb 14 00:00:00 2023 GMT
*  expire date: Mar 14 23:59:59 2024 GMT
*  subjectAltName: host "github.com" matched cert's "github.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]

...

* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55c127b9bb60)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
> GET /foo/bar/blob/master/model.tflite HTTP/2
> Host: github.com
> user-agent: curl/7.81.0
> accept: */*
> authorization: token <redacted Token>
> 

Not sure if it is relevant as I know very little about network security, but does gradle-download-task support this ca-certificate lookup, SSL, HTTP2, and TLSv1.2?

from gradle-download-task.

I see. The difference is that curl is using authorization: token ... instead of authorization: basic .... You've used the -H parameter to set the HTTP header accordingly (-H "Authorization: token $GITHUB_TOKEN"). Just do the same with gradle-download-task and you should be good to go:

task downloadModelFileRobot(type: Download) {
    header 'Authorization', 'token ' + System.getenv("GITHUB_TOKEN")
    src 'https://github.com/foo/bar'
    dest project.ext.ASSET_DIR + '/bar'
    overwrite false
}

from gradle-download-task.

Thanks for your ongoing support. I tried, and it still gave me the 404 error, but looking back at the curl results, it appears that also did not work (the contents of the file were simply "Not Found" rather than the actual binary). I should have verified that, sorry.

It appears this is not an issue with gradle-download-task so much as how GitHub is handling serving release assets in private/internal repos (from what I can tell). As a test, I did the following:

1. Upload the identical model.tflite file to a public repo as a part of a release. This is just a sandbox repo (so source code is completely unreleated)
2. Try to curl this public asset with: curl -o model.tflite -L https://github.com/topherbuckley/mavenTest/releases/download/0.1.1/model.tflite (this works as expected)
3. Try to curl the zipped source file from the Internal repo with curl -o tmp -H "Authorization: token $GITHUB_TOKEN" -L https://github.com/foo/bar/archive/refs/tags/0.1.1.zip. This also works
4. Try to curl the model.tflite file with idential syntax: curl -o model.tflite -H "Authorization: token $GITHUB_TOKEN" -L https://github.com/foo/bar/releases/download/0.1.1/model.tflite This changes the contents of the local model.tflite to "Not Found".
5. I can use the same link: https://github.com/foo/bar/releases/download/0.1.1/model.tflite in firefox, and it downloads just fine.

You can close this issue now (as I don't think this is specifically related to gradle-download-task) unless you are interested to follow my progress on this as it pertains to the use of gradle-download-task later.

from gradle-download-task.

If you are curious, this discusses this issue more detail, though I haven't found a place discussing this generalized to github, as the link above is a project discussion that is running into the same issue.

Essentially, I need to use the GitHub rest API to find an asset_id, then I can create a url to download that asset. I see no way of grabbing this asset_id other than using the rest API. Copying the syntax from a post within the link:

curl -v -L -H 'Accept: application/octet-stream' -H 'Authorization: token $GITHUB_TOKEN' https://api.github.com/repos/<owner>/<repo>/releases/assets/<asset_id>

You can find this asset_id via: curl -L -H "Accept: application/vnd.github+json" -H "Authorization: token $GITHUB_TOKEN" -H "X-GitHub-Api-Version: 2022-11-28" https://api.github.com/repos/<owner>/<repo>/releases/<release_id>/assets

And to get the release_id you need:
curl -L -H "Accept: application/vnd.github+json" -H "Authorization: token $GITHUB_TOKEN" -H "X-GitHub-Api-Version: 2022-11-28" https://api.github.com/repos/<owner>/<repo>/releases

This is all per the GitHub rest API

At this point, it might be easier to just use the GitHub cli to downloaded these. Maybe you can come up with a more elegant solution to incorporate into gradle-download-task, or just wait on the GitHub team to make this easier.

The octokit github account has a few open/closed issues surrounding this topic. This one has a decent boilerplate code in (Javascript?) to automate the process of what I described in the steps above.

from gradle-download-task.

Finally got something to work. I'm not a groovy or gradle guru by any means, so if you have tips for touching it up, I'm glad to hear them

def String getAssetID(String token, String owner, String repo, String tag_name, String asset_name){
    def jsonFile = new ByteArrayOutputStream()
    String path = "https://api.github.com/repos/${owner}/${repo}/releases".toString()
    def asset_info
    def asset_id
    exec {
        commandLine 'curl', '-L', '-H', 'Accept: application/vnd.github+json', '-H', "Authorization: token $token", '-H', 'X-GitHub-Api-Version: 2022-11-28', path

        standardOutput = jsonFile
    }
    def jsonContent = new JsonSlurper().parseText(jsonFile.toString())
    def release_info = null
    for (def obj : jsonContent) {
        if (obj.tag_name == tag_name) {
            release_info = obj
            def assets
            if (release_info != null) {
                assets = release_info.assets
//                    println("assets: $assets")
                if (assets != null) {
                    for (def asset : assets) {
                        if (asset.name == asset_name) {
                            asset_info = asset
                            println("found asset $asset_name")
                            break
                        }
                    }
                    if (asset_info != null){
                        asset_id = asset_info.id
                        println("asset_id: " + asset_id)
                        return asset_id
                    }else {
                        println "Asset with $asset_name not found"
                    }
                } else {
                    println "Release with tag $tag_name has no assets"
                }
            } else {
                println "Release with tag $tag_name not found"
            }
            break
        }
    }
}

task downloadModelFileRobot(type: Download) {
    doFirst{
        String token = System.getenv("GITHUB_TOKEN")
        String owner = 'foo'
        String repo = 'bar'
        String tag_name="0.1.1"
        String asset_name = "model.tflite"
        String asset_id = getAssetID(token, owner, repo, tag_name, asset_name)
        println("asset_id: " + asset_id)
        header 'Authorization', 'token ' + System.getenv("GITHUB_TOKEN")
        src "https://api.github.com/repos/$owner/$repo/releases/assets/$asset_id"
        dest project.ext.ASSET_DIR + '/model.tflite'
        overwrite false
    }
}

from gradle-download-task.

As an aside, and to anyone wanting to do something similar on a different JSON structure, I can also get this to work with a much simpler call to gh (github cli)

With:

task downloadModelFileRobot(type: Exec) {
    ignoreExitValue true
    commandLine 'echo', '$GITHUB_TOKEN', '|','gh', 'auth', 'login', '--with-token'
    commandLine 'gh', 'release', 'download', '0.1.1', '-p', 'model.tflite', '-R', 'foo/bar '-D', project.ext.ASSET_DIR, '--skip-existing'
}

from gradle-download-task.

Yep, sorry for the spamming comments on this. Hope it helps someone else in the future though. :) Thanks for your help as well!

from gradle-download-task.