Non-encoding of URL queries causing Regex robots.txt rule matches to be missed about protego HOT 3 OPEN

Sounds like a good fix.

I do wonder what the right behavior should be if the rule is /en-uk/*q%3D*relevance* instead of /en-uk/*q=*relevance*. We should find out, in case we need to adjust the fix accordingly (i.e. we may need to decode rules as well).

from protego.

It looks like Google published a standard last September, which seems to detail how the encoding/decoding should be handled. Now it is a matter of actually following it.

from protego.

I think as long as there is consistency in how rules and URLs are encoded, then decoding won't be necessary. I couldn't find any official word on the right behaviour, but I'd expect (for example) = and %3D to be treated as the same thing. So if we encode the rule and the URL, it won't matter how each is specified. Right?
Protego already decodes rules and URLs before encoding them, presumably to ensure consistency in formatting.

Your point made me look into the code a little further, and I notice that the "query" part is also not touched for a rule string. E.g.:

rule = Disallow: /en-uk/*?*specialFeatures=*|*|*
parts = ParseResult(scheme='', netloc='', path='/en-uk/*', params='', query='*specialFeatures=*|*|*', fragment='')
encoded rule = /en-uk/*?*specialFeatures=*|*|*

Asterisks (as well as a few other special characters) are ignored, but | should be encoded in theory - and they would be if in the "path" part. If we modify the code again to encode "query" like we did with URLs, we get this:

rule = Disallow: /en-uk/*?*specialFeatures=*|*|*
parts = ParseResult(scheme='', netloc='', path='/en-uk/*', params='', query='*specialFeatures%3D*%7C*%7C*', fragment='')
encoded rule = /en-uk/*?*specialFeatures%3D*%7C*%7C* 

In this way, any URL, regardless of formatting, should match a corresponding rule, regardless of formatting. Because they'll all be encoded in the same way.

parts is created with six.moves.urllib.parse.urlparse(), and there seems to be some inconsistency in how this library parses a URL. For example, from the rule in my previous post: q=*relevance* is found in the "path" part, when it should be in "query". Which is likely the source of the original problem...

So perhaps it's also worth considering encoding the "params" and "fragment" parts in the same way. They probably wouldn't actually be present in any rules, but urlparse may parse the URL incorrectly and use these parts too.

from protego.