Code Monkey home page Code Monkey logo

dy-danmu's Introduction

DY-DanMu

斗鱼弹幕抓取

架构.png

具体思路

根据斗鱼开发文档 https://open.douyu.com/source/api/63
进行websocket端抓取
协议.png
消息长度:4 字节小端整数,表示整条消息(包括自身)长度(字节数)。
消息长度出现两遍,二者相同。
消息类型:2 字节小端整数,表示消息类型。取值如下:
689 客户端发送给弹幕服务器的文本格式数据。
690 弹幕服务器发送给客户端的文本格式数据。
加密字段:暂时未用,默认为 0。保留字段:暂时未用,默认为 0。
websocket标准请查询
https://datatracker.ietf.org/doc/rfc6455/
中文版讲解
https://segmentfault.com/a/1190000005680323

##具体服务为三个
1. spider端:
连接websocket后实时获取弹幕信息通过gorutine连接rpc服务器进行存储
2. RPC-server端:
连接数据库进行数据存储 查询工作
3. Web端:
实现api接口进行参数解析 连接rpc进行查询

协议.png

由于数据量剧增 新增cache层(redis缓存)。

dy-danmu's People

Contributors

xiaxichen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

dy-danmu's Issues

提交一处bug

感谢作者的项目加速了我学习golang的进程,在测试中发现了一处bug,还望及时修复。
心跳包的发送周期是45秒是对的,但是那里时间单位错了。

time.Sleep(time.Microsecond * 500)

应该是毫秒级time.Sleep(time.Millisecond * 500)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.