yddchsc / eastmoney Goto Github PK

View Code? Open in Web Editor NEW

5.0 5.0 2.0 416 KB

a crawler by scrapy of eastmoney.com

Python 99.38% Shell 0.62%

eastmoney's People

Contributors

Stargazers

Watchers

Forkers

guotechfin

eastmoney's Issues

能否解释一下为什么采用ghost?

虽然用ghost的话ajax解决起来舒服很多, 但是用了ghost我就无法假设每次parse的负载是接近的了, 这个项目最终会应用于scrapy-redis类似的分布式架构, parse负载不均衡就要多写好些行代码做负载均衡,

所以说啊, 我想知道你是怎么做出这个决定的, 是传统方法遇到了不可逾越的困难?

为ItemPipeline类添加from_crawler, from_settings工厂方法

比如说

class GeguyaowenPipeline(object):
    def __init__(self, mongo_server, mongo_port):
            #建立连接之类的
            pass

     @classmethod
      def from_crawler(cls, crawler):
             return cls.from_settings(crawler.settings)

       @classmethod
       def from_settings(cls, settings):
            mongo_server = settings.get('MONGO_SERVER','localhost')
            mongo_port = settings.get('MONGO_PORT',27017)
            return cls(mongo_server,mongo_port)

参考: http://doc.scrapy.org/en/latest/topics/item-pipeline.html

Recommend Projects