Code Monkey home page Code Monkey logo

zsaim / videocrawlerengine Goto Github PK

View Code? Open in Web Editor NEW
445.0 16.0 88.0 6.41 MB

起源于旧项目爱奇艺解析器(iqiyi-parser)在开发、维护和扩展的过程中遇到的一些问题,而实现的一个基于任务流式的可视化爬虫引擎。引擎的执行单元是节点。脚本节点(script)作为根节点来完成对节点和流程的描述,经由任务节点(task)解析流程描述并生成节点的执行队列,最后交由工作者执行池处理。整个过程可视可控,所有节点处理器都以插件的形式导入,以最大程度实现易扩展性。

Python 65.23% HTML 0.12% JavaScript 13.96% Vue 20.69%
python taskflow

videocrawlerengine's Issues

使用npm run dev命令报错

博主,您好!
我在运行您的项目的时候,按照你的要求使用npm run dev或者yarn dev,均产生如下错误:
Error: Cannot find module 'E:\python\VideoCrawlerEngine-master\app\html\build\webpack.dev.conf.js'
Require stack:

  • E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack\bin\convert-argv.js
  • E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack-dev-server\bin\webpack-dev-server.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
    at Function.Module._load (internal/modules/cjs/loader.js:725:27)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at requireConfig (E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack\bin\convert-argv.js:97:18)
    at E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack\bin\convert-argv.js:104:17
    at Array.forEach ()
    at module.exports (E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack\bin\convert-argv.js:102:15)
    at Object. (E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack-dev-server\bin\webpack-dev-server.js:223:50)
    at Module._compile (internal/modules/cjs/loader.js:1063:30) {
    code: 'MODULE_NOT_FOUND',
    requireStack: [
    'E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack\bin\convert-argv.js',
    'E:\python\VideoCrawlerEngine-master\app\html\node_modules\webpack-dev-server\bin\webpack-dev-server.js'
    ]
    }

我对前端不是很了解,也尝试解决过这个问题,但是都失败了,请问下这个是项目本身有问题,还是我的环境有问题呢?

[bug] 请求失败

我是搜【爱奇艺】找到原项目 iqiyi-parser 然后跳到这个项目的
clone 到本地装好依赖运行,打开网页,输入爱奇艺的视频网址
然后弹出请求失败的对话框,如下,是此项目不支持爱奇艺还是有 bug?😂
20201122154208

其实觉得还是命令行稍微好点

首先 感谢您愿意将如此棒的项目分享出来
并抱歉我没有这个精力和您一起为此项目做出贡献
由于工作原因我仅能等待您的更新与修复
从iqiyi-parser到VideoCrawlerEngine
虽然后者因为某些限制我没有尝试过运行
但根据目前更新时间来看
下载,转码这些请求完全可以我们自己实现
您辛苦做这么多实际上有的地方也让自己很头疼不是吗
如果能使用命令行 类似 yt-dl 那样(不用他是因为不支持VIP最高画质)
我们仅需要一个解析出的m3u8链接
以现在的网络 在该CDN过期前可以下载完整部电视剧
且ffmpeg -i 可以直接下载和转码
至于批量的话notepad++处理字符串然后作为batch运行起来也没有那么不如意
所以觉得作者您的这些辛苦都有些白费
如果按我说的 只做命令行
那么您将会获得更多的时间 也没有那么累
我们也可以获得更快的更新和BUG修复
不是吗

安装环境运行报错

使用的系统是centos7.64位,Python版本3.6.8。
已经按照文档要求先使用pip install -r requirements.txt安装相关环境,但是最后Python mian.py的时候报错。
File "main.py", line 2, in
from script import init_scripts
File "/home/VideoCrawlerEngine-master/script/init.py", line 5, in
from utils import utility_package, split_name_version
File "/home/VideoCrawlerEngine-master/utils.py", line 12, in
from debugger import dbg
File "/home/VideoCrawlerEngine-master/debugger.py", line 3, in
from contextvars import ContextVar
ModuleNotFoundError: No module named 'contextvars'

怎么运行?

我直接clone下来,还需要做其他配置吗?
python main.py 后没有任何输出

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.