Code Monkey home page Code Monkey logo

zsaim / videocrawlerengine Goto Github PK

View Code? Open in Web Editor NEW
442.0 16.0 88.0 6.41 MB

起源于旧项目爱奇艺解析器(iqiyi-parser)在开发、维护和扩展的过程中遇到的一些问题,而实现的一个基于任务流式的可视化爬虫引擎。引擎的执行单元是节点。脚本节点(script)作为根节点来完成对节点和流程的描述,经由任务节点(task)解析流程描述并生成节点的执行队列,最后交由工作者执行池处理。整个过程可视可控,所有节点处理器都以插件的形式导入,以最大程度实现易扩展性。

Python 65.23% HTML 0.12% JavaScript 13.96% Vue 20.69%
python taskflow

videocrawlerengine's People

Contributors

zsaim avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

videocrawlerengine's Issues

怎么运行?

我直接clone下来,还需要做其他配置吗?
python main.py 后没有任何输出

使用npm run dev命令报错

博主,您好!
我在运行您的项目的时候,按照你的要求使用npm run dev或者yarn dev,均产生如下错误:
Error: Cannot find module 'E:\python\VideoCrawlerEngine-master\app\html\build\webpack.dev.conf.js'
Require stack:

  • E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack\bin\convert-argv.js
  • E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack-dev-server\bin\webpack-dev-server.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
    at Function.Module._load (internal/modules/cjs/loader.js:725:27)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at requireConfig (E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack\bin\convert-argv.js:97:18)
    at E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack\bin\convert-argv.js:104:17
    at Array.forEach ()
    at module.exports (E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack\bin\convert-argv.js:102:15)
    at Object. (E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack-dev-server\bin\webpack-dev-server.js:223:50)
    at Module._compile (internal/modules/cjs/loader.js:1063:30) {
    code: 'MODULE_NOT_FOUND',
    requireStack: [
    'E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack\bin\convert-argv.js',
    'E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack-dev-server\bin\webpack-dev-server.js'
    ]
    }

我对前端不是很了解,也尝试解决过这个问题,但是都失败了,请问下这个是项目本身有问题,还是我的环境有问题呢?

安装环境运行报错

使用的系统是centos7.64位,Python版本3.6.8。
已经按照文档要求先使用pip install -r requirements.txt安装相关环境,但是最后Python mian.py的时候报错。
File "main.py", line 2, in
from script import init_scripts
File "/home/VideoCrawlerEngine-master/script/init.py", line 5, in
from utils import utility_package, split_name_version
File "/home/VideoCrawlerEngine-master/utils.py", line 12, in
from debugger import dbg
File "/home/VideoCrawlerEngine-master/debugger.py", line 3, in
from contextvars import ContextVar
ModuleNotFoundError: No module named 'contextvars'

其实觉得还是命令行稍微好点

首先 感谢您愿意将如此棒的项目分享出来
并抱歉我没有这个精力和您一起为此项目做出贡献
由于工作原因我仅能等待您的更新与修复
从iqiyi-parser到VideoCrawlerEngine
虽然后者因为某些限制我没有尝试过运行
但根据目前更新时间来看
下载,转码这些请求完全可以我们自己实现
您辛苦做这么多实际上有的地方也让自己很头疼不是吗
如果能使用命令行 类似 yt-dl 那样(不用他是因为不支持VIP最高画质)
我们仅需要一个解析出的m3u8链接
以现在的网络 在该CDN过期前可以下载完整部电视剧
且ffmpeg -i 可以直接下载和转码
至于批量的话notepad++处理字符串然后作为batch运行起来也没有那么不如意
所以觉得作者您的这些辛苦都有些白费
如果按我说的 只做命令行
那么您将会获得更多的时间 也没有那么累
我们也可以获得更快的更新和BUG修复
不是吗

[bug] 请求失败

我是搜【爱奇艺】找到原项目 iqiyi-parser 然后跳到这个项目的
clone 到本地装好依赖运行,打开网页,输入爱奇艺的视频网址
然后弹出请求失败的对话框,如下,是此项目不支持爱奇艺还是有 bug?😂
20201122154208

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.