Comments (9)
ncw
commented on June 10, 2024
Hmm, interesting!
Can you check if rclone does the same thing with incorrect credentials with AWS S3?
I suspect this will need to be fixed in the upstream library.
from rclone.
static-moonlight
commented on June 10, 2024
I don't have an AWS S3 available, but when I replace the "rclone serve s3" server with MinIO and send the same request with a wrong password, I see this response:
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<Resource>/</Resource>
<RequestId>17C4FD2A69658918</RequestId>
<HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId>
</Error>... and the rclone s3 client immediately fails with an error:
2024/04/10 20:01:56 ERROR : : error listing: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
status code: 403, request id: 17C4FD2A69658918, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
2024/04/10 20:01:56 Failed to lsf with 2 errors: last error was: error in ListJSON: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
status code: 403, request id: 17C4FD2A69658918, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
I think it's safe to say that "rclone serve s3" is creating an invalid xml response message.
The rclone client is fine, i think.
from rclone.
ncw
commented on June 10, 2024
Please give this a try
v1.67.0-beta.7876.5c6c39510.fix-7749-s3-serve-error on branch fix-7749-s3-serve-error (uploaded in 15-30 mins)
from rclone.
static-moonlight
commented on June 10, 2024
Sorry, that didn't fix it :( The error is still there, exactly the same.
$ ./rclone --version
rclone v1.67.0-beta.7876.5c6c39510.fix-7749-s3-serve-error
- os/version: debian 12.1 (64 bit)
- os/kernel: 6.1.0-20-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.2
- go/linking: static
- go/tags: none
from rclone.
ncw
commented on June 10, 2024
This fixes the XML for me
Before (the master branch)
2024/04/18 11:59:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/04/18 11:59:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/04/18 11:59:03 DEBUG : HTTP REQUEST (req 0xc000b88500)
2024/04/18 11:59:03 DEBUG : GET / HTTP/1.1
Host: localhost:8080
User-Agent: rclone/v1.67.0-beta.7896.2063e017d
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240418T105903Z
Accept-Encoding: gzip
2024/04/18 11:59:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/04/18 11:59:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/04/18 11:59:03 DEBUG : HTTP RESPONSE (req 0xc000b88500)
2024/04/18 11:59:03 DEBUG : HTTP/1.1 403 Forbidden
Content-Length: 236
Content-Type: text/xml; charset=utf-8
Date: Thu, 18 Apr 2024 10:59:03 GMT
<?xml version="1.0" encoding="UTF-8"?>
<errorResponse><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message></errorResponse>
After (with the fix above)
2024/04/18 12:00:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/04/18 12:00:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/04/18 12:00:20 DEBUG : HTTP REQUEST (req 0xc000d10100)
2024/04/18 12:00:20 DEBUG : GET / HTTP/1.1
Host: localhost:8080
User-Agent: rclone/v1.67.0-beta.7876.5c6c39510.fix-7749-s3-serve-error
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240418T110020Z
Accept-Encoding: gzip
2024/04/18 12:00:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/04/18 12:00:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/04/18 12:00:20 DEBUG : HTTP RESPONSE (req 0xc000d10100)
2024/04/18 12:00:20 DEBUG : HTTP/1.1 403 Forbidden
Content-Length: 236
Content-Type: text/xml; charset=utf-8
Date: Thu, 18 Apr 2024 11:00:20 GMT
<?xml version="1.0" encoding="UTF-8"?>
<ErrorResponse><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message></ErrorResponse>
I tested this with running this in one terminal
rclone serve -vv --auth-key user,pass s3 /tmp/src/
And this in another (note the incorrect secret_access_key)
rclone lsf -vv ':s3,provider=Rclone,access_key_id=user,secret_access_key=passX,endpoint="http://localhost:8080":' --dump bodies
What problem are you having and how can I replicate it? Preferably with terminal commands like the above not with containers.
from rclone.
static-moonlight
commented on June 10, 2024
I'm gonna test it again. Maybe I did something wrong.
from rclone.
static-moonlight
commented on June 10, 2024
Sorry, but this really didn't fix the problem. The problem is that the error response has a wrong root xml element. This patch basically changed the wrong element name into another wrong element name.
- was (1.66):
<errorResponse> - is now (1.67 patch):
<ErrorResponse> - expected (S3 specification):
<Error>
from rclone.
ncw
commented on June 10, 2024
Ah, sorry I mis-understood the problem.
Can you try this please
v1.67.0-beta.7876.2dc6016de.fix-7749-s3-serve-error on branch fix-7749-s3-serve-error (uploaded in 15-30 mins)
PS Incorrect root XML nodes are very common - I found a bug exactly like that in Google Cloud Storage's S3 implementation recently!
from rclone.
static-moonlight
commented on June 10, 2024
That looks better :)
- I can verify that the xml message in the response now has the correct root xml element.
- rclone (as a client) now fails instantly (before, it was retrying over and over again for about a minute)
2024/04/23 10:09:03 ERROR : : error listing: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method. status code: 403, request id: , host id: 2024/04/23 10:09:03 Failed to lsf with 2 errors: last error was: error in ListJSON: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method. status code: 403, request id: , host id:
I would say this pretty much fixes the problem.
Thanks you for the patch!
The only thing I just noticed it that the "request id" and "host id" (in the log above) have no values. This is probably because the corresponding <RequestId> and <HostId> xml elements are missing in the response. I leave it to you though to decide if that is an issue or not ;)
from rclone.
Related Issues (20)
- Backup dir cache from ram to disk and reload into ram on reboot
- Autodetect terminal width or add option to --progress to specify width HOT 2
- uloz backend, copyto doesn't work (Error 60001 (422): Validation failed) HOT 8
- Web Dav mount files appear as 1999-09-04 date HOT 1
- armv5tejl arch Illegal instruction HOT 3
- Options recover and max-lock missing in cmd/bisync/rc.go HOT 3
- Cannot access SMB share via Kerberos HOT 2
- Add "cryptdecode" function as a flag (--cryptdecode remote:) to copy/move/etc. commands?
- rclone lsf with --include argument can not find the Correct file HOT 5
- WebDAV: Indicate final file size when doing chunk upload
- please ignore HOT 3
- Support Permanent delete in OneDrive/Sharepoint HOT 5
- Recursively lock/unlock files from remote:path in Google Drive HOT 3
- Feature request: Hasher, copy hashes on server-side copy (maybe simple copy too)
- --files-from doesn't seem to play nicely with --links HOT 2
- `rclone ls` does not exit with an error code on various cloud backends when file doesn't exist HOT 2
- Hasher: trust cached sum
- Git backend? HOT 2
- Windows: Destination Encoding characters are improperly escaped HOT 4
- OAuth tokens fail to refresh when they are provided as a JSON blob (for native-app clients) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rclone.