python3webspider / scrapyredisbloomfilter Goto Github PK

View Code? Open in Web Editor NEW

173.0 3.0 52.0 35 KB

Scrapy Redis Bloom Filter

Python 100.00%

scrapyredisbloomfilter's Introduction

Scrapy-Redis-BloomFilter

This is a package for supporting BloomFilter of Scrapy-Redis.

Installation

You can easily install this package with pip:

pip install scrapy-redis-bloomfilter

Dependency:

Scrapy-Redis >= 0.6.8

Usage

Add this settings to settings.py:

# Use this Scheduler, if your scrapy_redis version is <= 0.7.1
SCHEDULER = "scrapy_redis_bloomfilter.scheduler.Scheduler"

# Ensure all spiders share same duplicates filter through redis
DUPEFILTER_CLASS = "scrapy_redis_bloomfilter.dupefilter.RFPDupeFilter"

# Redis URL
REDIS_URL = 'redis://localhost:6379'

# Number of Hash Functions to use, defaults to 6
BLOOMFILTER_HASH_NUMBER = 6

# Redis Memory Bit of Bloom Filter Usage, 30 means 2^30 = 128MB, defaults to 30
BLOOMFILTER_BIT = 10

# Persist
SCHEDULER_PERSIST = True

Test

Here is a test of this project, usage:

git clone https://github.com/Python3WebSpider/ScrapyRedisBloomFilter.git
cd ScrapyRedisBloomFilter/test
scrapy crawl test

Note: please change REDIS_URL in settings.py.

Spider like this:

from scrapy import Request, Spider

class TestSpider(Spider):
    name = 'test'
    base_url = 'https://www.baidu.com/s?wd='

    def start_requests(self):
        for i in range(10):
            url = self.base_url + str(i)
            yield Request(url, callback=self.parse)

        # Here contains 10 duplicated Requests
        for i in range(100):
            url = self.base_url + str(i)
            yield Request(url, callback=self.parse)

    def parse(self, response):
        self.logger.debug('Response of ' + response.url)

Result like this:

{'bloomfilter/filtered': 10, # This is the number of Request filtered by BloomFilter
 'downloader/request_bytes': 34021,
 'downloader/request_count': 100,
 'downloader/request_method_count/GET': 100,
 'downloader/response_bytes': 72943,
 'downloader/response_count': 100,
 'downloader/response_status_count/200': 100,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2017, 8, 11, 9, 34, 30, 419597),
 'log_count/DEBUG': 202,
 'log_count/INFO': 7,
 'memusage/max': 54153216,
 'memusage/startup': 54153216,
 'response_received_count': 100,
 'scheduler/dequeued/redis': 100,
 'scheduler/enqueued/redis': 100,
 'start_time': datetime.datetime(2017, 8, 11, 9, 34, 26, 495018)}

scrapyredisbloomfilter's People

Contributors

Stargazers

Watchers

Forkers

acefei yangshimin mpkato junqiangle alchemy2011 jun-ge zbooa clanstyles hangyuge dex7erhan 123weizheng wgwjifeng ii0 fengmstrong zuiwengf houzhong kingking888 yxlspider lovezeropython liujiali vickey-wu allenshen321 tjp40922 aaronchiu2017 songhao8080 jxsylar gabeochieng javajiang daijingyu123 aweng126 xiaowxhang dorothylyly luodongfu skyjustblue liangnet zanachka chenglong-s dao258 ykallan hxlszxy pylemonorg vonkonyoung bluebad xsfhacg tieyongjie yisongcai xy424 kanadeblisst githubforliuqiao aivr1 850312766 hjc1985

scrapyredisbloomfilter's Issues

url num: 720,000
memory: 3200m
======settings.py
SCHEDULER = "scrapy_redis_bloomfilter.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis_bloomfilter.dupefilter.RFPDupeFilter"
BLOOMFILTER_HASH_NUMBER = 6
BLOOMFILTER_BIT = 30
SCHEDULER_PERSIST = True
REDIS_PARAMS = {'password':'xxx','db':2}
How is this memory so high? The bloomer algorithm should not account for too much memory.Is that normal?

how could i settings request_depth_max?

老哥方便留个联系方式吗?有个问题想请教下

或者你加我QQ348211764,谢谢.

有个弊端把start_urls里面的链接也给去重了

起始链接被去重后面就没法抓取了，望改进

当BLOOMFILTER_BIT设置为30时，起不到去重效果。

> fp = '75d6587d87b3f4f3aa574b33dbd69ceeb9eafe7b'
> bf = BloomFilter(server, 'ldy4', 30, 6)
> bf.exists(fp)
0
> bf.insert(fp)
> bf.exists(fp)
0

当BLOOMFILTER_BIT 设置为22时没有问题，不知道是什么原因

有个问题想了解下.

原先项目设置的256M,如果要在配置里增加容量,之前去重的数据还会保留吗
如果不能保留有什么办法..谢谢

运行测试报错 ModuleNotFoundError: No module named 'tests.settings'

我是把包下载下来，然后执行 scrapy crawl test 命令，会报以下错误：

\ScrapyRedisBloomFilter-master\tests> scrapy crawl test
Traceback (most recent call last):
  File "c:\programdata\anaconda3\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\programdata\anaconda3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\Scripts\scrapy.exe\__main__.py", line 7, in <module>
  File "c:\programdata\anaconda3\lib\site-packages\scrapy\cmdline.py", line 114, in execute
    settings = get_project_settings()
  File "c:\programdata\anaconda3\lib\site-packages\scrapy\utils\project.py", line 69, in get_project_settings
    settings.setmodule(settings_module_path, priority='project')
  File "c:\programdata\anaconda3\lib\site-packages\scrapy\settings\__init__.py", line 287, in setmodule
    module = import_module(module)
  File "c:\programdata\anaconda3\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'tests.settings'

目录如下：

│  .gitignore
│  README.md
│  requirements.txt
│  setup.py
│
├─scrapy_redis_bloomfilter
│      bloomfilter.py
│      defaults.py
│      dupefilter.py
│      scheduler.py
│      __init__.py
│      __version__.py
│
└─tests
    │  scrapy.cfg
    │
    └─tests
        │  items.py
        │  middlewares.py
        │  pipelines.py
        │  settings.py
        │  __init__.py
        │
        └─spiders
                test.py
                __init__.py
                test.py
                __init__.py

为什么呢？