View Code? Open in Web Editor
NEW
A go practice for creating a crawler from scratch
gocrawler's Introduction
- Install dependencies
make install
- Build
make build
- Run
MODE=<enviroment> config=<config file path> <inital URL>
- Wait it finish and open the sitemap file, done
- go ~1.9.3
- glide ~0.13.1
- make 3.81
- A configuration file required during executing, the format can be found inside
./config/prod/parameters.yaml
.
- Executing environment is required, need to be set as an environment variable, the variable name is
MODE
. Default environment is taken as debug
.
- You need to define the sitemap save path on your machine.
gocrawler's People
Contributors
Watchers