Bad request after redirect about gradle-download-task HOT 15 CLOSED

Hi Michel,

I've just tested your latest snapshot version and the download now succeeds. As far as I'm concerned you can go ahead and publish the new version and close this issue.

Thanks to you and the jetbrains team for figuring this one out.

from gradle-download-task.

Thanks a lot, @eghobo! That's valuable information. I was able to reproduce the issue.

It seems other HTTP client such as xh strip sensitive headers on redirect (https://github.com/ducaale/xh/blob/master/src/redirect.rs#L95-L101) but Apache HttpClient does not. If I understand HttpClient's source code correctly, it copies the original request on redirect, including all headers! It does clear the authentication cache indeed, but since you're sending the header directly, it's not in said cache and thus won't get cleared.

Now I need to figure out, if there is a way to remove sensitive headers on redirect like xh does, or if I need to implement a custom authentication scheme so the default mechanisms of HttpClient take effect. The issue with the latter is that BitBucket, for example, sends a 404 if the request is not pre-authenticated. A custom authentication scheme would require a 401 and some kind of WWW-Authenticate header in the server's response. However, maybe I can force pre-authentication if an Authorization header is provided in the request. I need to test this.

The good news is: now that I'm able to reproduce the issue, I can fix it. Just give me a day or two. I will get back to you as soon as possible.

from gradle-download-task.

@michel-kraemer it also works for me with Bitbucket Repository Access Token

from gradle-download-task.

Thanks for the debug log. I assume you're using the latest version of gradle-download-task?

from gradle-download-task.

Yes, 5.5.0
Thank you for your quick reaction

from gradle-download-task.

I've just published a snapshot version 5.6.0-SNAPSHOT. It adds a new property validateStatus, which can be used to allow certain HTTP status codes. In your case, you can use it to download the body of the error response you're getting from your server. Just add the following line to your task configuration:

validateStatus { it == 400 }

Here's a complete example:

tasks.register<Download>("downloadFile") {
    src("https://example.com/404")
    dest(layout.buildDirectory)
    overwrite(true)
    validateStatus { it < 500 }
}

After this, your destination file should contain the server error message. This might help us to identify the root cause. Let me know what you get. Thanks!

In order to use the snapshot, you have to add the following block to your settings.gradle file:

pluginManagement {
    repositories {
        maven {
            url "https://oss.sonatype.org/content/groups/public"
        }
        gradlePluginPortal()
    }
}

from gradle-download-task.

Hi Michel,

Your snapshot worked perfectly, this is the response I got:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>Bearer *****</ArgumentValue><RequestId>1P2J4GTTPB8V9VVE</RequestId><HostId>GHY43plCmoKjd+7Avies81s8lQ95HfnWm61AMMqriCnhf/6ZTva2jEUhiPTLiEIi0WCEGbSUmPM=</HostId></Error>

Is this something you can work with?

Thank you for your effort.

Best regards,
Diederik Faber

from gradle-download-task.

Perhaps the request has to be signed:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/RESTAuthentication.html

You can define your own headers with the header property.

from gradle-download-task.

We're already providing authentication using the header property. The error says that only one authentication mechanism is allowed, but the second mechanism is added somewhere in the redirect. So that is out of my hands. If I leave out my own authentication header, then the first request is immediately denied and the redirect does not occur at all.

I've asked for support from jetbrains on this as well. I guess I have to wait for their feedback.
Your support so far is very much appreciated.

from gradle-download-task.

@michel-kraemer I suspect that the issue is following when HTTP Client sends a request to the host1 which redirects to s3buckethost with presigned URL gradle download task preserves Authorization header from the initial request and sends it to the s3buckethost.

See the related issues in ktorio/ktor#1467

from gradle-download-task.

@dtretyakov Thanks for the pointer. I had the same thought. If that's the case then it might be an issue with Apache's HttpClient. I will try to figure out if there is a workaround or if the authentication headers can be reset after the redirect (just like it says in the Ktor thread).

from gradle-download-task.

I've asked for support from jetbrains on this as well. I guess I have to wait for their feedback.

@diederikfaber Sounds good. It'd be great if you could keep me posted.

from gradle-download-task.

@michel-kraemer we have a similar problem with https://api.bitbucket.org/ and Repository Access Tokens, when we pass header 'Authorization: Bearer <repository_access_token> . Bitbucket redirects download requests to AWS S3, which doesn't accept this Authorization header.

from gradle-download-task.

I think I've fixed the issue. It would be great if someone could confirm that. I've uploaded a new snapshot 5.6.0-SNAPSHOT (see instructions above). You may have to call gradle with --refresh-dependencies to download the latest version of the snapshot.

After the fix has been confirmed, I can publish the new version.

Thank you so much for your help!!

from gradle-download-task.

I've just published version 5.6.0! Thanks to all in this thread for the pointers and the help with testing!

from gradle-download-task.