I'm trying to use lxml.Cleaner without parsing response multiple times: <div class

This is fixed since <a class="issue-link js-issue-link" data-error-text="Failed to loa

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Selector.root is not an instance of lxml.html.HtmlElement even if parser is html about parsel HOT 7 CLOSED

scrapy commented on July 26, 2024

Selector.root is not an instance of lxml.html.HtmlElement even if parser is html

from parsel.

Comments (7)

eliasdorneles commented on July 26, 2024 1

This is fixed since #63:

>>> import parsel
>>> sel = parsel.Selector(u'<html><body><h1>oi</h1></body></html>')
>>> type(sel.root)
lxml.html.HtmlElement

Thanks @kmike !

from parsel.

redapple commented on July 26, 2024

@kmike , there was some discussion in the past about moving to lxml.html.HtmlParser
scrapy/scrapy#559 (comment)
but it was left at that, needing performance comparison

from parsel.

rmax commented on July 26, 2024

@kmike @redapple Related #41

from parsel.

kmike commented on July 26, 2024

Another unscientific benchmark results (Python 3.5.1, OS X, latest lxml): https://gist.github.com/kmike/af647777cef39c3d01071905d176c006. It seems there is a 1-5% penalty in using lxml.html.HTMLParser, based on ~3700 random pages. My vote is to set HTMLParser as a default.

from parsel.

kmike commented on July 26, 2024

Another way to look at it: additional time required for using HTMLParser is 0.0001s per web page on average (3700 pages, total parsing time is increased by 0.4s).

from parsel.

eliasdorneles commented on July 26, 2024

yeah, let's just do it! 👍

from parsel.

eliasdorneles commented on July 26, 2024

@kmike what do you think? #54

from parsel.

Recommend Projects