twiny / spidy Goto Github PK

Is your feature request related to a problem? Please describe.
I am not sure if its -u url1.com url2.com or -u url1.com,url2.com or -u url1.com -u url2.com

I am sorry im not familiar enough with go yet to figure it out. But if you explain ill submit a pull request to update the readme

Could one crawl domains from a given csv file instead of giving right from the prompt?

Describe the bug
The crawler marks domains like justin.tv (old twitch.tv) as available when they are not. Inside the results .csv

To Reproduce

1. install
2. run on any website
3. wait to see results

Expected behavior
.tv domains are correctly categorized.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):
tested on windows 10, linux & alpinejs

Spidy aborts with "Killed". Do you have a clue why?

Thanks!

Is it possible to implement Open Page Rank?

Is it possible to only list available domains in the result csv?

Encountered this error with go version go1.15.6 linux/amd64 on Ubuntu 20.04.1 LTS and the following command line:

curl -v -silent -l https://twitter.com --stderr - | awk '/^content-security-policy:/' | grep -Eo "[a-zA-Z0-9./?=_-]*" | sed -e '/\./!d' -e '/[^A-Za-z0-9._-]/d' -e 's/^\.//' | sort -u | httpx -o spidy/input.txt && spidy -config spidy/setting.yaml

Welcome, Spidy is running.
false schema.org
false googletagmanager.com
false giphy.com
false facebook.net
false google-analytics.com
false vine.co
false twitter.com
false t.co
false doubleclick.net
false vineapp.com
false nytimes.com
false twimg.com
false pscp.tv
false google.com
false apple.com
false bit.ly
false happs.tv
false n.tw
false jquery.com
false wordpress.com
false pocoo.org
false apache.org
false jquery.org
false gruntjs.com
false emberjs.com
false github.com
false mattt.me
false bower.io
false stylus-lang.com
false ogp.me
false ethicspointvp.com
false jamsadr.com
false dataprotection.ie
false privacyshield.gov
panic: send on closed channel

goroutine 172 [running]:
github.com/superiss/spidy/crawler.(*Spider).extract(0xc000276600, 0xc000586000, 0x3b7d, 0x3e00)
        /root/go/pkg/mod/github.com/superiss/[email protected]/crawler/crawler.go:268 +0x105
github.com/superiss/spidy/crawler.(*Spider).Run.func4(0xc000276600, 0xc00006cea0)
        /root/go/pkg/mod/github.com/superiss/[email protected]/crawler/crawler.go:352 +0x71
created by github.com/superiss/spidy/crawler.(*Spider).Run
        /root/go/pkg/mod/github.com/superiss/[email protected]/crawler/crawler.go:350 +0x21b

setting.yaml:

Engine:
  worker: 100
  parallel: 10
  depth: 10
  urls: ./spidy/input.txt
  proxies: []
  #tlds: [com, net, io, co, ly, me, us, at, st, so]
  tlds: []
  random_delay: 5s
  timeout: 30s

httpx can be found here: https://github.com/projectdiscovery/httpx

Is it possible to only crawl deeper and not upwards in hierarchy or sideways?

So when i provide the url https://test.com/test1/test2/test3/ that the crawler only crawls everything under /test3/ but not things under /test2/ or higher?