Scrapy-CVE-CNVD

基于scrapy框架，获取每日最新的CVE和CNVD漏洞，主要目的用于安全研究，获取最新安全咨询，无任何漏洞利用细节，严禁用于非法活动。

为了保证信息的易读性，CVE漏洞详细信息主要参照CNNVD官网信息，可能导致部分CVE无法被收录。更多信息可以通过关注CVE官方的Twitter获取。

Start

Docker-compose部署

修改crontabfile配置邮箱等信息

PATH=/opt/someApp/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# 个人的账号信息
[email protected]
EMAIL_PASS=xxxxxxxxxxxxxx
[email protected]
# 10分钟执行一次
*/10 * * * * cd /root/vuls && ./run.sh >> /var/log/cron.log

构建（网络问题多重试几次）

cd Scrapy-CVE-CNVD
docker-compose build

运行（首次运行需等待一段时间）

docker-compose up -d

本地部署

下载项目

git clone https://github.com/ttonys/Scrapy-CVE-CNVD

安装依赖(python3环境)

pip install -r requirements.txt

redis安装----docker

docker pull redis
docker run -itd --name redis-test -p 6379:6379 redis

redis安装----本机

参照网上教程

修改配置，Scrapy-CVE-CNVD/OnePiece/pipelines.py文件36-38行，使用自己注册163邮箱，以及所获取的授权码（非个人密码）

fromaddr = '[email protected]'
password = 'xxxxxxxxxxxxxx'
toaddrs = '[email protected]'

修改配置，Scrapy-CVE-CNVD/OnePiece/settings.py文件，配置个人的redis服务器ip、端口和密码

REDIS_HOST = '127.0.0.1'
REDIS_PORT = 6379
REDIS_PARAMS = {
'password': '123456',
}

修改配置，Scrapy-CVE-CNVD/run.sh中，将路径修改为自己本机的路径，参照如下：

#!/bin/bash
cd /root/Scrapy-CVE-CNVD
/usr/local/bin/scrapy crawl all-redis

测试，将Scrapy-CVE-CNVD路径下文件改为755权限（chmod -R 755 Scrapy-CVE-CNVD），运行./run.sh，查看是否收到邮件通知

添加定时任务，注意使用绝对路径，按照本机实际路径填写，参考如下（每天下午2点半执行）：

30 14 * * * /root/Scrapy-CVE-CNVD/run.sh

Source

国家信息安全漏洞共享平台

https://www.cnvd.org.cn/

国家信息安全漏洞库

http://www.cnnvd.org.cn/

today-cve

https://cassandra.cerias.purdue.edu/CVE_changes/today.html

cve官网

https://cve.mitre.org/

Demo

数据保存

邮件内容

builtins.TypeError: _findCaller() takes from 1 to 2 positional arguments but 3 were given

报错关键字

builtins.TypeError: _findCaller() takes from 1 to 2 positional arguments but 3 were given

报错内容

└─# ./run.sh 
Unhandled error in Deferred:
Temporarily disabling observer LegacyLogObserverWrapper(<bound method PythonLoggingObserver.emit of <twisted.python.log.PythonLoggingObserver object at 0x7fd62e5e2f40>>) due to exception: [Failure instance: Traceback: <class 'TypeError'>: _findCaller() takes from 1 to 2 positional arguments but 3 were given
/usr/lib/python3/dist-packages/twisted/internet/defer.py:953:__del__
/usr/lib/python3/dist-packages/twisted/logger/_logger.py:270:critical
/usr/lib/python3/dist-packages/twisted/logger/_logger.py:144:emit
--- <exception caught here> ---
/usr/lib/python3/dist-packages/twisted/logger/_observer.py:131:__call__
/usr/lib/python3/dist-packages/twisted/logger/_legacy.py:93:__call__
/usr/lib/python3/dist-packages/twisted/python/log.py:595:emit
/usr/lib/python3/dist-packages/twisted/logger/_legacy.py:154:publishToNewObserver
/usr/lib/python3/dist-packages/twisted/logger/_stdlib.py:115:__call__
/usr/lib/python3.9/logging/__init__.py:1500:log
/usr/lib/python3.9/logging/__init__.py:1565:_log
]
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 953, in __del__
    log.critical("Unhandled error in Deferred:",
  File "/usr/lib/python3/dist-packages/twisted/logger/_logger.py", line 270, in critical
    self.emit(LogLevel.critical, format, **kwargs)
  File "/usr/lib/python3/dist-packages/twisted/logger/_logger.py", line 144, in emit
    self.observer(event)
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/twisted/logger/_observer.py", line 131, in __call__
    observer(event)
  File "/usr/lib/python3/dist-packages/twisted/logger/_legacy.py", line 93, in __call__
    self.legacyObserver(event)
  File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 595, in emit
    _publishNew(self._newObserver, eventDict, textFromEventDict)
  File "/usr/lib/python3/dist-packages/twisted/logger/_legacy.py", line 154, in publishToNewObserver
    observer(eventDict)
  File "/usr/lib/python3/dist-packages/twisted/logger/_stdlib.py", line 115, in __call__
    self.logger.log(
  File "/usr/lib/python3.9/logging/__init__.py", line 1500, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib/python3.9/logging/__init__.py", line 1565, in _log
    fn, lno, func, sinfo = self.findCaller(stack_info, stacklevel)
builtins.TypeError: _findCaller() takes from 1 to 2 positional arguments but 3 were given


Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/scrapy/crawler.py", line 192, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/scrapy/crawler.py", line 196, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
    _inlineCallbacks(None, g, status)
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python3.9/dist-packages/scrapy/crawler.py", line 89, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
redis.exceptions.ResponseError: AUTH <password> called without any password configured for the default user. Are you sure your configuration is correct?

Temporarily disabling observer LegacyLogObserverWrapper(<bound method PythonLoggingObserver.emit of <twisted.python.log.PythonLoggingObserver object at 0x7fd62e5e2f40>>) due to exception: [Failure instance: Traceback: <class 'TypeError'>: _findCaller() takes from 1 to 2 positional arguments but 3 were given
/usr/lib/python3/dist-packages/twisted/internet/defer.py:962:__del__
/usr/lib/python3/dist-packages/twisted/logger/_logger.py:190:failure
/usr/lib/python3/dist-packages/twisted/logger/_logger.py:144:emit
--- <exception caught here> ---
/usr/lib/python3/dist-packages/twisted/logger/_observer.py:131:__call__
/usr/lib/python3/dist-packages/twisted/logger/_legacy.py:93:__call__
/usr/lib/python3/dist-packages/twisted/python/log.py:595:emit
/usr/lib/python3/dist-packages/twisted/logger/_legacy.py:154:publishToNewObserver
/usr/lib/python3/dist-packages/twisted/logger/_stdlib.py:115:__call__
/usr/lib/python3.9/logging/__init__.py:1500:log
/usr/lib/python3.9/logging/__init__.py:1565:_log
]
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 962, in __del__
    log.failure(format,
  File "/usr/lib/python3/dist-packages/twisted/logger/_logger.py", line 190, in failure
    self.emit(level, format, log_failure=failure, **kwargs)
  File "/usr/lib/python3/dist-packages/twisted/logger/_logger.py", line 144, in emit
    self.observer(event)
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/twisted/logger/_observer.py", line 131, in __call__
    observer(event)
  File "/usr/lib/python3/dist-packages/twisted/logger/_legacy.py", line 93, in __call__
    self.legacyObserver(event)
  File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 595, in emit
    _publishNew(self._newObserver, eventDict, textFromEventDict)
  File "/usr/lib/python3/dist-packages/twisted/logger/_legacy.py", line 154, in publishToNewObserver
    observer(eventDict)
  File "/usr/lib/python3/dist-packages/twisted/logger/_stdlib.py", line 115, in __call__
    self.logger.log(
  File "/usr/lib/python3.9/logging/__init__.py", line 1500, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib/python3.9/logging/__init__.py", line 1565, in _log
    fn, lno, func, sinfo = self.findCaller(stack_info, stacklevel)
builtins.TypeError: _findCaller() takes from 1 to 2 positional arguments but 3 were given

Python版本

python3                                                                                                                                               1 ⨯
Python 3.9.0+ (default, Oct 20 2020, 08:43:38) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

scrapy

└─# cat /usr/local/bin/scrapy
#!/usr/bin/python3
# -*- coding: utf-8 -*-

ttonys / scrapy-cve-cnvd Goto Github PK

scrapy-cve-cnvd's Introduction

Scrapy-CVE-CNVD

Start

Docker-compose部署

本地部署

Source

Demo

scrapy-cve-cnvd's People

Contributors

Stargazers

Watchers

Forkers

scrapy-cve-cnvd's Issues

报错关键字

报错内容

Python版本

scrapy

Recommend Projects

Recommend Topics

Recommend Org