Comments (3)
jasonz1360
commented on June 2, 2024
能否实现多个任务共用一个git项目，这样一个任务一个任务的拉太耗时间了
from crawlab.
jasonz1360
commented on June 2, 2024
git pull 怎么拉取代码，都有文件是删除状态
from crawlab.
jasonz1360
commented on June 2, 2024
找到个解决办法，挂载共用目录，这样git只需要sync一次，还省力
举例方法：
volumes:
- "/opt/.crawlab/master:/root/.crawlab" # 持久化 crawlab 元数据
- "/opt/crawlab/master:/data" # 持久化 crawlab 数据
- "/var/crawlab/log:/var/log/crawlab" # 持久化 crawlab 任务日志
- "/opt/finbase-crawlab:/root/***:rw" # 持久化 crawlab 代码
然后在爬虫的命令里加上root/***开头
from crawlab.
Related Issues (20)
- Installing Software Dependencies HOT 2
- Install crawlab with spesific python version HOT 1
- Crawlab AI Assistant
- crawlab-server 占用内存过高问题 HOT 3
- The background page can execute any command HOT 1
- 点击取消后，节点任务并未实际结束的问题
- 希望增加爬虫任务失败重试执行次数设置
- 希望未来可以开放通知的api接口
- container crash (Panic) HOT 1
- Data Sources HOT 1
- crawlab 重启之后，重启前 pending 的任务无法继续执行(After crawlab restart, tasks pending before restart cannot continue) HOT 6
- SeaweedFS定期出现8888端口无法连接，爬虫的文件也丢失。 HOT 6
- 重置密码未在文档中看到
- 随机节点偶发调度到多slave上
- 当数据源选择MySQL时，存储失败 HOT 7
- Crawlab File System Optimization
- 磁盘被占满了
- Local files getting deleted after git pull
- 有arm镜像吗？
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from crawlab.