amasser / scrapyd-go Goto Github PK

This project forked from alash3al/scrapyr

0.0 1.0 0.0 19 KB

a simple & tiny scrapy clustering solution, considered a drop-in replacement for scrapyd

License: Apache License 2.0

Dockerfile 1.13% Go 98.87%

scrapyd-go's Introduction

scrapyd-go

an drop-in replacement for scrapyd that is more easy to be scalable and distributed on any number of commodity machines with no hassle, each scrapyd-go instance is a stateless microservice, all instances must be connected to the same redis server, redis is used as a ceneralized registry system for all instances, so each instance se what others see.

Why

scrapyd isn't bad, but it is very stateful, it isn't that easy to deploy it in a destributed environment like k8s, as well as I wanted to add more features, so I started this project as a drop-in replacement for scrapyd but writing in modern & scalable environment like go for restful server and redis as centeralized registry.

TODOs

Configurations

scrapyd-go configs are just simple command line flags

  -dir string
        the directory to use for local caching (default ".scrapyd-go")
  -listen string
        the address to bind to (default ":6800")
  -max2keep int
        the maximum jobs/logs to keep in memory (default 1000000)
  -poll int
        time in millisecond between each poll operation from queue(s) (default 10)
  -python string
        the python binary to use (default "python3")
  -redis string
        the redis server address (default "redis://:somepass@localhost:6379/1")
  -sync int
        time in seconds between each sync operation (default 15)
  -workers int
        the maximum workers count (default cpu-cores-count)

Installation

binary : go to releases page and download your os based release
docker: $ docker pull alash3al/scrapyd-go
source: $ go get github.com/alash3al/scrapyd-go

Running

binary: $ ./scrapyd_bin_file -redis redis://localhost:6379/1
docker: $ docker run --link SomeRedisServerContainer -p 6800:6800 alash3al/scrapyd-go -redis redis://SomeRedisServerContainer:6379/1
source: $ scrapyd-go -redis redis://localhost:6379/1