s0rg / crawley Goto Github PK
View Code? Open in Web Editor NEWThe unix-way web crawler
License: MIT License
The unix-way web crawler
License: MIT License
Hello, when I use this command:
crawley -depth -1 -dirs only -cookie "phpbb3_XXXXXX_sid=XXXXXX; phpbb3_XXXXXX_u=XXXXXX; phpbb3_XXXXXX_k=XXXXXX;" "https://XXXXXX.net/viewtopic.php?p=5018859" > urls.txt
crawley scrapes the login page of the forum and not the thread I selected, crawley doesn't return any errors
2023/03/20 16:08:58 [*] config: workers: 4 depth: -1 delay: 150ms
2023/03/20 16:08:58 [*] crawling url: https://XXXXXX.net/viewtopic.php?p=5018859
2023/03/20 16:09:00 [*] complete
I tried with different user agents and headers, every time the result is the same: login page of the forum. The cookies are ok, I copied them using EditThisCookie googlechrome extension. I'm not using any vpn/proxy.
I tested also other forums, the result is always the same, I can't login.
Do you know if there is a problem loading cookies?
Is their a way to use credentials / a session cookie to scrap a site on which I am logged in ?
Please add support for subdomains recursive search.
For example: if I run crawley on https://abc.com, I can add a flag (e.g. -subdomains) to allow crawling also for subdomains under abc.com (e.g. www.abc.com or somesub.abc.com)
Thanks!
I want to scrape this forum www.invitehawk.com but I get this deadlock error. I know it's a cookie error. In this case I can scrape even without cookies but I'd like to know che correct cookie syntax. I'm using netscape cookie format.
โ crawley --cookie invitehawk_ck.txt -depth -1 -dirs only "https://www.invitehawk.com/topic/126147-tracker-review-index-sorted-by-category/" | grep -E '.*20[0-9][0-9]-review' > url.list.ih.txt
2023/03/19 10:17:11 [*] config: workers: 4 depth: -1 delay: 150ms
2023/03/19 10:17:11 [*] crawling url: https://www.invitehawk.com/topic/126147-tracker-review-index-sorted-by-category/
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0x0?)
/opt/hostedtoolcache/go/1.20.0/x64/src/runtime/sema.go:62 +0x27
sync.(*WaitGroup).Wait(0xc0000b01e0?)
/opt/hostedtoolcache/go/1.20.0/x64/src/sync/waitgroup.go:116 +0x4b
github.com/s0rg/crawley/pkg/crawler.(*Crawler).close(0xc0000b2840)
/home/runner/work/crawley/crawley/pkg/crawler/crawler.go:201 +0x65
panic({0x6c1960, 0xc0000c20c0})
/opt/hostedtoolcache/go/1.20.0/x64/src/runtime/panic.go:884 +0x213
github.com/s0rg/crawley/pkg/client.parseOne({0x7ffc5e21c176?, 0x11?})
/home/runner/work/crawley/crawley/pkg/client/cookie.go:35 +0x13d
github.com/s0rg/crawley/pkg/client.prepareCookies({0xc00009e430?, 0x1, 0xc000170000?})
/home/runner/work/crawley/crawley/pkg/client/cookie.go:17 +0x13c
github.com/s0rg/crawley/pkg/client.New({0xc0000cc280, 0x3e}, 0x4, 0x0, {0xc00009e440, 0x1, 0x1}, {0xc00009e430, 0x1, 0x1})
/home/runner/work/crawley/crawley/pkg/client/http.go:57 +0x1f8
github.com/s0rg/crawley/pkg/crawler.(*Crawler).Run(0xc0000b2840, {0x7ffc5e21c19d, 0x50}, 0x70ee68)
/home/runner/work/crawley/crawley/pkg/crawler/crawler.go:99 +0x29b
main.crawl({0x7ffc5e21c19d, 0x50}, {0xc0000b4200?, 0x0?, 0xc0000406c8?})
/home/runner/work/crawley/crawley/cmd/crawley/main.go:94 +0xe5
main.main()
/home/runner/work/crawley/crawley/cmd/crawley/main.go:235 +0x188
I tried several commands:
crawley -depth -1 -headless -cookie "phpbb3_ddu4final_k=XXXXX; phpbb3_ddu4final_sid=XXXXX; phpbb3_ddu4final_u=XXXXX;" "https://ddunlimited.net/viewtopic.php?p=5018859" > test.txt
crawley -depth -1 -headless -cookie "phpbb3_ddu4final_k=XXXXX;phpbb3_ddu4final_sid=XXXXX;phpbb3_ddu4final_u=XXXXX;" "https://ddunlimited.net/viewtopic.php?p=5018859" > test.txt
crawley -depth -1 -headless -cookie "phpbb3_ddu4final_k=XXXXX;" "https://ddunlimited.net/viewtopic.php?p=5018859" > test.txt
crawley -depth -1 -headless -cookie "phpbb3_ddu4final_k=XXXXX;" -cookie "phpbb3_ddu4final_sid=XXXXX;" -cookie "phpbb3_ddu4final_u=XXXXX;" "https://ddunlimited.net/viewtopic.php?p=5018859" > test.txt
crawley -cookie "phpbb3_ddu4final_k=XXXXX;" "https://ddunlimited.net/viewtopic.php?p=5018859" > test.txt
2023/03/29 23:36:18 [*] config: workers: 4 depth: -1 delay: 150ms
2023/03/29 23:36:18 [*] crawling url: https://ddunlimited.net/viewtopic.php?p=5018859
2023/03/29 23:36:18 cannot parse 'phpbb3_ddu4final_k=XXXXX; phpbb3_ddu4final_sid=XXXXX; phpbb3_ddu4final_u=XXXXX;' as cookie, expected format: 'key=value;' as in curl
2023/03/29 23:36:20 [*] complete
It works when I don't use semicolon:
crawley -depth -1 -headless -cookie "phpbb3_ddu4final_k=XXXXX" "https://ddunlimited.net/viewtopic.php?p=5018859" > test.txt
Crawley v1.5.12-a1f6de2 (archlinux)
Add flag to ignore scraping same URL but with different params.
For example, while scraping a website https://abc.com
, the flag will disable scraping both https://abc.com/something.php?lang=en
and https://abc.com/something.php?lang=ru
, since they are the same page but with different params.
Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.