Comments (6)
So in the new version of Scrapy, what method can replace response_httprepr()?
I did not find a corresponding replacement method in https://docs.scrapy.org/en/latest?
from scrapy.
There is no replacement. Scrapy was using that function to measure response length, but now it uses an alternative approach (with better performance), so the function is no longer needed by Scrapy itself, and will eventually be removed without a replacement.
from scrapy.
Oh, yes! I also need to calculate the length of the response as a reference for traffic consumption. Could you tell me if Scrapy currently employs a more efficient method for this? I would like to have a reference.
Looking forward to your reply!
from scrapy.
Or will future versions of Scrapy provide such a method as a generic utility class?
from scrapy.
There are no plans to provide a utility function for this, but you can see in https://github.com/scrapy/scrapy/pull/4972/files how we are doing the calculations now.
from scrapy.
Good, I understand, just use the following code reslen = len(response.body) + get_header_size(response.headers) + get_status_size(response.status) + 4
. Thank you so much!
from scrapy.
Related Issues (20)
- Implement get_import _path
- Replace urlparse with urlparse_cached where possible
- execution of asyncio.ensure_future(coro()) ignored on close_spider() pipelines call HOT 4
- Explanation of the robots.txt exclusion standard in DownloaderMiddleware.robotstxt.py HOT 1
- Use `defusedxml.xmlrpc`
- AttributeError: 'Decompressor' object has no attribute 'process' HOT 8
- Fix and re-enable `unnecessary-comprehension` and `use-dict-literal` pylint tags
- Investigate speeding up `MockServer()` HOT 3
- Proxy tests don't use tests/keys/mitmproxy-ca.pem HOT 2
- Add brotlicffi support
- scrapy.pqueues.ScrapyPriorityQueue HOT 8
- Remove tests/requirements.txt
- Add an extra-deps job for pypy
- Document the SpiderState extension
- WindowsRunSpiderCommandTest isn't skipped properly in the pinned envs
- More documentation needed about the robots.txt protocol HOT 3
- GZipPlugin does not work with S3 HOT 3
- AttributeError: 'Decompressor' object has no attribute 'process' HOT 1
- Handle robots.txt files not utf-8 encoded HOT 3
- SitemapSpider will ignore sitemap with URLs like https://website.com/filename.xml?from=7155352010944&to=7482320519360 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scrapy.