Code Monkey home page Code Monkey logo

dhtcrawler's Introduction

DHTCrawler

python 编写的DHT Crawler 网络爬虫,抓取DHT网络的磁力链接。

文件

collector.py dht网络爬虫脚本

抓取dht网络的磁力链接,使用 libtorrent 的python绑定库开发

collectord.py dht爬虫服务监控程序

启动并监控dht爬虫进程,在爬虫进程退出后重启启动爬虫,使用 twisted 开发

安装和使用方法

运行环境

运行的机器要能被外网访问,用来接收dht网络节点的信息,使用vps即可

  1. linux 服务器
  2. python 2.7.3
  3. 安装 libtorrent 的 python 绑定库
  4. 安装 twisted 网络库
  5. 开放防火墙的对应端口段,目前默认的是 32900--32920 (20是工作的p2p客户端数量)

运行方法

  1. 下载 collector.py collectord.py 文件到安装目录
  2. 脚本方式测试运行: python collector.py result.json collector.state
  3. 服务方式运行:twistd -y collectord.py

状态查看

  1. 通过 twistd 服务启动时,服务状态查看:telnet 127.0.0.1 31000
  2. 命令行或服务启动是,爬虫状态查看:cat collector.state

运行结果

result.json 是收集的磁力链接结果文件,json格式,key是资源的info hash,value是资源的热度

Go 开发的电商平台,欢迎合作

技术:golang、vue、微信小程序开发

  • 一群工程师开发的B2B、B2C电商平台
  • 目前客户月GMV百万级
  • 可以提供整套服务合作、或者整套源代码出售
  • 电商平台简介
  1. 海外版跨境电商h5、web平台
  2. 经销商车销平台(业务员下单,司机带货管理)
  3. 天阶高性能 B2B、B2C 电商平台,商城与管理后台完全分离,支持多公司多仓库加盟模式 支持高并发,集群水平扩展,微信小程序商城

golang 电商开发技术交流群

  • 欢迎大家交流电商开发技术
  • QQ群:226220067 加群请说明来自 github

dhtcrawler's People

Contributors

blueskyz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dhtcrawler's Issues

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.