Reported by kurtjx on Trac h

this is the same problem described at <a href="https://github.com/kennethreitz/request

pass tests <div class="snippet-clipboard-content notranslate position-relative ove

SgmlLinkExtractor mangles some url encoding (specifically "/" encoded) about scrapy HOT 4 CLOSED

scrapy commented on May 21, 2024

SgmlLinkExtractor mangles some url encoding (specifically "/" encoded)

from scrapy.

Comments (4)

dangra commented on May 21, 2024

just tried this, it logs a warning now and it is caused by restrict_xpaths argument, looks same bug than #199

$ scrapy shell 'http://www.last.fm/music/AC%252FDC/+images'
2013-01-29 12:51:24-0200 [scrapy] INFO: Scrapy 0.17.0 started (bot: scrapybot)
...
>>> from scrapy.contrib.linkextractors import sgml
>>> sgml.SgmlLinkExtractor(restrict_xpaths=('//a[@class="nextlink"]')).extract_links(response)
/home/daniel/src/scrapy/scrapy/link.py:16: UserWarning: Do not instantiate Link objects with unicode urls. Assuming utf-8 encoding (which could be wrong)
  warnings.warn("Do not instantiate Link objects with unicode urls. " \
[Link(url='http://www.last.fm/music/AC/DC/+images?page=2', text=u'Next', fragment='', nofollow=False)]

from scrapy.

dangra commented on May 21, 2024

this is actually a different bug, happens that canonicalize_url is unquoting the url path converting %2f into / and that is a wrong url for last.fm.

from scrapy.

dangra commented on May 21, 2024

this is the same problem described at https://github.com/kennethreitz/requests/pull/273

and wikipedia explain it more clearly what characters are reserved and must be quoted in a path http://en.wikipedia.org/wiki/Percent-encoding

from scrapy.

dangra commented on May 21, 2024

pass tests

scrapy.tests.test_contrib_linkextractors
  LinkExtractorTestCase
    test_base_url ...                                                      [OK]
    test_basic ...                                                         [OK]
    test_extraction_encoding ...                                           [OK]
    test_link_nofollow ...                                                 [OK]
    test_link_text_wrong_encoding ...                                      [OK]
    test_matches ...                                                       [OK]
  SgmlLinkExtractorTestCase
    test_base_url_with_restrict_xpaths ...                                 [OK]
    test_deny_extensions ...                                               [OK]
    test_encoded_url ...                                                   [OK]
    test_encoded_url_in_restricted_xpath ...                               [OK]
    test_extraction ...                                                    [OK]
    test_extraction_using_single_values ...                                [OK]
    test_matches ...                                                       [OK]
    test_process_value ...                                                 [OK]
    test_restrict_xpaths ...                                               [OK]
    test_restrict_xpaths_concat_in_handle_data ...                         [OK]
    test_restrict_xpaths_encoding ...                                      [OK]
    test_urls_type ...                                                     [OK]
scrapy.tests.test_utils_url
  UrlUtilsTest
    test_canonicalize_url ...                                              [OK]
    test_url_is_from_any_domain ...                                        [OK]
    test_url_is_from_spider ...                                            [OK]
    test_url_is_from_spider_class_attributes ...                           [OK]
    test_url_is_from_spider_with_allowed_domains ...                       [OK]
    test_url_is_from_spider_with_allowed_domains_class_attributes ...      [OK]
Doctest: scrapy.utils
  url
    escape_ajax ...                                                        [OK]

-------------------------------------------------------------------------------
Ran 25 tests in 0.031s

PASSED (successes=25)

from scrapy.

Recommend Projects