Welcome, Spidy is running.
false schema.org
false googletagmanager.com
false giphy.com
false facebook.net
false google-analytics.com
false vine.co
false twitter.com
false t.co
false doubleclick.net
false vineapp.com
false nytimes.com
false twimg.com
false pscp.tv
false google.com
false apple.com
false bit.ly
false happs.tv
false n.tw
false jquery.com
false wordpress.com
false pocoo.org
false apache.org
false jquery.org
false gruntjs.com
false emberjs.com
false github.com
false mattt.me
false bower.io
false stylus-lang.com
false ogp.me
false ethicspointvp.com
false jamsadr.com
false dataprotection.ie
false privacyshield.gov
panic: send on closed channel
goroutine 172 [running]:
github.com/superiss/spidy/crawler.(*Spider).extract(0xc000276600, 0xc000586000, 0x3b7d, 0x3e00)
/root/go/pkg/mod/github.com/superiss/[email protected]/crawler/crawler.go:268 +0x105
github.com/superiss/spidy/crawler.(*Spider).Run.func4(0xc000276600, 0xc00006cea0)
/root/go/pkg/mod/github.com/superiss/[email protected]/crawler/crawler.go:352 +0x71
created by github.com/superiss/spidy/crawler.(*Spider).Run
/root/go/pkg/mod/github.com/superiss/[email protected]/crawler/crawler.go:350 +0x21b
Engine:
worker: 100
parallel: 10
depth: 10
urls: ./spidy/input.txt
proxies: []
#tlds: [com, net, io, co, ly, me, us, at, st, so]
tlds: []
random_delay: 5s
timeout: 30s