rclone copy with http remote fails with HTTP Error: 404 Not Found when URL contains question mark about rclone HOT 10 CLOSED

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.67

Thank you to @KDreynolds for implementing it.

I'll close this issue now.

( Note @KDreynolds that the words in the commit have to be fixes #xxx to auto close an issue )

from rclone.

I can begin working on this if no one else is.

from rclone.

% rclone copy --dump bodies --http-no-head -vv --http-url 'https://www.google.com' ':http:search?q=rclone' /tmp

I think this command invocation is wrong...

I think it should be

rclone copy --dump bodies --http-no-head -vv --http-url 'https://www.google.com/search?q=rclone' ':http:' /tmp

Parts of the URL need to go into the --http-url - things after the :http: are expected to be directory of file names which will need the ? URL encoding.

from rclone.

I see no reason why anything beyond scheme + hostname needs to go into the --http-url param. Seems like a misinterpretation of the W3C URL spec. For example, in my use case the part that goes into --http-url is scheme + host name and the rest of the URLs all come from the contents of --files-from. This allow to access a list of arbitrary URLs on that host, not just URLs from a given fixed prefix within a host.

from rclone.

I see no reason why anything beyond scheme + hostname needs to go into the --http-url param.

That is how the http backend was designed.

Remember (in normal use) it reads links from web pages and it ignores all URLs with ? parameters. Any escaped ? are treated as part of the path. This is so you can have file names with ? in.

For example, in my use case the part that goes into --http-url is scheme + host name and the rest of the URLs all come from the contents of --files-from. This allow to access a list of arbitrary URLs on that host, not just URLs from a given fixed prefix within a host.

That is an interesting use case for the http backend, but it isn't how it was designed to work. It was designed to read the file listings generated by webservers.

Perhaps we need a new flag --http-no-escape which means don't escape the URL metacharacters in the path names? Or maybe a specific flag --http-urls-from meaning don't parse any web pages, just use these URLs.

from rclone.

Perhaps we need a new flag --http-no-escape which means don't escape the URL metacharacters in the path names?

Yes, that would be perfect!

from rclone.

@whoschek do you want to have a go at adding such a flag?

from rclone.

@whoschek do you want to have a go at adding such a flag?

Thanks, but I'm afraid I don't have the cycles in the foreseeable future :-(
Looks like @KDreynolds would be up for it, though.

from rclone.

Finally got around to taking a crack at this one. PR #7777

from rclone.

I always find new ways to do something small wrong lolsad, thank you!

from rclone.