viknim / dark_web_scraping Goto Github PK
View Code? Open in Web Editor NEWThis repository contains scrapers programs to scrape hacking forums from Dark web
This repository contains scrapers programs to scrape hacking forums from Dark web
Excuse me, I want to ask how to get the data after running the script like this. Below is my running log。
Looking forward to your reply
➜ tutorial scrapy crawl raidForums -o items.json 2020-05-08 18:03:29 [scrapy.utils.log] INFO: Scrapy 2.0.1 started (bot: tutorial) 2020-05-08 18:03:29 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.7.3 (default, Sep 5 2019, 17:14:41) - [Clang 11.0.0 (clang-1100.0.33.8)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1f 31 Mar 2020), cryptography 2.9, Platform Darwin-19.2.0-x86_64-i386-64bit 2020-05-08 18:03:29 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor 2020-05-08 18:03:29 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'tutorial', 'COOKIES_ENABLED': False, 'FEED_FORMAT': 'json', 'FEED_URI': 'items.json', 'NEWSPIDER_MODULE': 'tutorial.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['tutorial.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' '(KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36'} 2020-05-08 18:03:29 [scrapy.extensions.telnet] INFO: Telnet Password: b92d496e2ae53916 2020-05-08 18:03:30 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats'] 2020-05-08 18:03:30 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2020-05-08 18:03:30 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2020-05-08 18:03:30 [scrapy.middleware] INFO: Enabled item pipelines: [] 2020-05-08 18:03:30 [scrapy.core.engine] INFO: Spider opened 2020-05-08 18:03:30 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2020-05-08 18:03:30 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2020-05-08 18:03:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://raidforums.com/robots.txt> (referer: None) 2020-05-08 18:03:32 [protego] DEBUG: Rule at line 1 without any user agent to enforce it on. 2020-05-08 18:03:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://raidforums.com/> (referer: None) 2020-05-08 18:03:33 [scrapy.core.engine] INFO: Closing spider (finished) 2020-05-08 18:03:33 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 600, 'downloader/request_count': 2, 'downloader/request_method_count/GET': 2, 'downloader/response_bytes': 25416, 'downloader/response_count': 2, 'downloader/response_status_count/200': 2, 'elapsed_time_seconds': 3.750242, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2020, 5, 8, 10, 3, 33, 862219), 'log_count/DEBUG': 3, 'log_count/INFO': 10, 'memusage/max': 60833792, 'memusage/startup': 60833792, 'response_received_count': 2, 'robotstxt/request_count': 1, 'robotstxt/response_count': 1, 'robotstxt/response_status_count/200': 1, 'scheduler/dequeued': 1, 'scheduler/dequeued/memory': 1, 'scheduler/enqueued': 1, 'scheduler/enqueued/memory': 1, 'start_time': datetime.datetime(2020, 5, 8, 10, 3, 30, 111977)} 2020-05-08 18:03:33 [scrapy.core.engine] INFO: Spider closed (finished) ➜ tutorial scrapy crawl raidForums -o items.json 2020-05-08 19:37:57 [scrapy.utils.log] INFO: Scrapy 2.0.1 started (bot: tutorial) 2020-05-08 19:37:57 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.7.3 (default, Sep 5 2019, 17:14:41) - [Clang 11.0.0 (clang-1100.0.33.8)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1f 31 Mar 2020), cryptography 2.9, Platform Darwin-19.2.0-x86_64-i386-64bit 2020-05-08 19:37:57 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor 2020-05-08 19:37:57 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'tutorial', 'COOKIES_ENABLED': False, 'FEED_FORMAT': 'json', 'FEED_URI': 'items.json', 'NEWSPIDER_MODULE': 'tutorial.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['tutorial.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' '(KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36'} 2020-05-08 19:37:57 [scrapy.extensions.telnet] INFO: Telnet Password: 07e1db1788fa0e06 2020-05-08 19:37:57 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats'] 2020-05-08 19:37:57 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2020-05-08 19:37:57 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2020-05-08 19:37:57 [scrapy.middleware] INFO: Enabled item pipelines: [] 2020-05-08 19:37:57 [scrapy.core.engine] INFO: Spider opened 2020-05-08 19:37:57 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2020-05-08 19:37:57 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2020-05-08 19:37:59 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://raidforums.com/robots.txt> (referer: None) 2020-05-08 19:37:59 [protego] DEBUG: Rule at line 1 without any user agent to enforce it on. 2020-05-08 19:38:00 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://raidforums.com/> (referer: None) 2020-05-08 19:38:00 [scrapy.core.engine] INFO: Closing spider (finished) 2020-05-08 19:38:00 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 600, 'downloader/request_count': 2, 'downloader/request_method_count/GET': 2, 'downloader/response_bytes': 25188, 'downloader/response_count': 2, 'downloader/response_status_count/200': 2, 'elapsed_time_seconds': 3.278753, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2020, 5, 8, 11, 38, 0, 853012), 'log_count/DEBUG': 3, 'log_count/INFO': 10, 'memusage/max': 61059072, 'memusage/startup': 61059072, 'response_received_count': 2, 'robotstxt/request_count': 1, 'robotstxt/response_count': 1, 'robotstxt/response_status_count/200': 1, 'scheduler/dequeued': 1, 'scheduler/dequeued/memory': 1, 'scheduler/enqueued': 1, 'scheduler/enqueued/memory': 1, 'start_time': datetime.datetime(2020, 5, 8, 11, 37, 57, 574259)} 2020-05-08 19:38:00 [scrapy.core.engine] INFO: Spider closed (finished)
Hi,
Hope you are all well !
I tried to run some scripts and it seems that some packages are missing.
Traceback (most recent call last):
File "bestBlackHatSpider.py", line 12, in <module>
from ..items import BestblackhatforumItem, ReplyItems
ValueError: Attempted relative import in non-package
Is it possible to fix it ?
Cheers,
X
Traceback (most recent call last):
File "C:\Users\admin\Desktop\Saurabh\osint\dark web\Dark_Web_Scraping-master\bestBlackHatSpider.py", line 12, in
from ..items import BestblackhatforumItem, ReplyItems
ImportError: attempted relative import with no known parent package
Hey, I was wondering how are you bypassing RF's recaptcha!
Thanks.
Hello, I will report an error when using Raidforum.py
Traceback (most recent call last):
File "raidForums.py", line 12, in <module>
from ..items import RaidforumsItem, ReplyItems
ValueError: attempted relative import beyond top-level package
Looking forward to your reply
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.