Comments (3)
Sounds like a good fix.
I do wonder what the right behavior should be if the rule is /en-uk/*q%3D*relevance*
instead of /en-uk/*q=*relevance*
. We should find out, in case we need to adjust the fix accordingly (i.e. we may need to decode rules as well).
from protego.
It looks like Google published a standard last September, which seems to detail how the encoding/decoding should be handled. Now it is a matter of actually following it.
from protego.
I think as long as there is consistency in how rules and URLs are encoded, then decoding won't be necessary. I couldn't find any official word on the right behaviour, but I'd expect (for example) =
and %3D
to be treated as the same thing. So if we encode the rule and the URL, it won't matter how each is specified. Right?
Protego already decodes rules and URLs before encoding them, presumably to ensure consistency in formatting.
Your point made me look into the code a little further, and I notice that the "query" part is also not touched for a rule string. E.g.:
rule = Disallow: /en-uk/*?*specialFeatures=*|*|*
parts = ParseResult(scheme='', netloc='', path='/en-uk/*', params='', query='*specialFeatures=*|*|*', fragment='')
encoded rule = /en-uk/*?*specialFeatures=*|*|*
Asterisks (as well as a few other special characters) are ignored, but |
should be encoded in theory - and they would be if in the "path" part. If we modify the code again to encode "query" like we did with URLs, we get this:
rule = Disallow: /en-uk/*?*specialFeatures=*|*|*
parts = ParseResult(scheme='', netloc='', path='/en-uk/*', params='', query='*specialFeatures%3D*%7C*%7C*', fragment='')
encoded rule = /en-uk/*?*specialFeatures%3D*%7C*%7C*
In this way, any URL, regardless of formatting, should match a corresponding rule, regardless of formatting. Because they'll all be encoded in the same way.
parts
is created with six.moves.urllib.parse.urlparse(), and there seems to be some inconsistency in how this library parses a URL. For example, from the rule in my previous post: q=*relevance*
is found in the "path" part, when it should be in "query". Which is likely the source of the original problem...
So perhaps it's also worth considering encoding the "params" and "fragment" parts in the same way. They probably wouldn't actually be present in any rules, but urlparse
may parse the URL incorrectly and use these parts too.
from protego.
Related Issues (14)
- Colons in file names prevent installation on NTFS HOT 2
- Accept robots.txt as bytes
- Disallowing / does not work when the target URL path is missing
- Document Python version support in the comparison table of the README
- Automatically deploy new Protego releases to conda-forge HOT 1
- Protego differs from reppy in handling of wildcards for GET-params HOT 4
- six usage HOT 1
- Update for RFC 9309 Robots Exclusion Protocol
- python-protego fails to build with Python 3.12: AttributeError: 'TestProtego' object has no attribute 'assertEquals'.
- Support for visit-time HOT 1
- Cannot fetch non-disallowed domain HOT 1
- URL starting with double-slash are misinterpreted HOT 1
- Use `pyre2` as optional dependency for RegExp speedup.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from protego.