elasticdeeplearning / edl Goto Github PK
View Code? Open in Web Editor NEWElastic Deep Learning for deep learning framework on Kubernetes
License: Apache License 2.0
Elastic Deep Learning for deep learning framework on Kubernetes
License: Apache License 2.0
the test includings:
Build images with Dockerfile;
Push images to docker hub;
Pull the images built above and run the EDL controller in k8s cluster;
Run the training job in k8s cluster to test the ASGD;
需要添加超时失败容错
Exception in thread Thread-3:
Traceback (most recent call last):
File "/root/paddlejob/workspace/env_run/python-2.7.14/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/root/paddlejob/workspace/env_run/python-2.7.14/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/root/paddlejob/workspace/env_run/python-2.7.14/lib/python2.7/site-packages/paddle_edl/distill/redis/client.py", line 95, in _heartbeat
msg = self._recv_msg()
File "/root/paddlejob/workspace/env_run/python-2.7.14/lib/python2.7/site-packages/paddle_edl/distill/redis/client.py", line 55, in _recv_msg
head = self.client.recv(self.HEAD_SIZE)
error: [Errno 110] Connection timed out
When I tried to run distributed CTR training on K8S Cluster. There is mechanical to collect all pservers' IP and port. But it seems that the concatenation of IP and port looks weird.
initially it is 172.20.1.69:30236, 172.20.1.70:30237, which are pserver 1 and pserver 2's IP.
but then it displays 172.20.1.69:30236, 172.20.1.70.30237:30236. the pserver 2 has two ports.
Please complete the transfer of edl repo to its own GitHub org located in: https://github.com/elasticdeeplearning/.
Here are the steps:
Thank you.
Ibrahim
Hello, I've been trying to run the resnet + imagenet demo shown here https://github.com/elasticdeeplearning/edl/tree/develop/example/demo/collective for several days now but with no success. I've tried doing this on my local machine by pip installing paddle_edl
into a conda environment and all associated requirements in addition to trying with the recommended docker image:
docker pull hub.baidubce.com/paddle-edl/paddle_edl:latest-cuda9.0-cudnn7
nvidia-docker run -name paddle_edl hub.baidubce.com/paddle-edl/paddle_edl:latest-cuda9.0-cudnn7 /bin/bash
I'm having the suspicion that this demo is outdated and I was wondering if it could be updated or explained in more detail so that I can get it working. If the demo is still in fact working, can someone run the demo using the expected docker image and let me know of the exact steps to replicate it.
I've tried running the demo using several different combinations of steps but here is what I'm doing in general.
Reproduction Steps:
edl/example/demo/collective
PADDLE_EDL_IMAGENET_PATH
, PADDLE_EDL_FLEET_CHECKPOINT_PATH
and PADDLE_JOBSERVER
./start_job_server.sh
./start_job_client.sh
Some issues that I have faced so far:
train.txt
, test.txt
or val.txt
for the imagenet dataset and have errors using mine with the edl demo. What is the expected preprocessing strategy that edl uses for their imagenet
dataset and how is it structured so I can use my own imagenet
dataset.src_dir=../../collective/resnet50
. There must have been some directories moved around as this is not the correct pathway to the resnet files.not ready endpoints:['127.0.0.1:8073', '127.0.0.1:8075', '127.0.0.1:8077', '127.0.0.1:8079', '127.0.0.1:8081', '127.0.0.1:8083', '127.0.0.1:8085']
server not ready, wait 3 sec to retry...
nohup python -u paddle_edl.demo.collective.job_server_demo
should use an -m
flag instead of -u
as paddle_edl
cannot be found otherwise
System information:
Thanks and looking forward to demoing the project!
We need an EDL tutorial to introduce how to use EDL on a Kubernetes cluster.
Fix unit test under python3
test_distill_reader.sh
test_redis_distill_reader.sh
To my knowledge, edl should rely on the capability of PaddlePaddle workload. In other words, PaddlePaddle must has the ability of elastic learning.
If my understanding is correct, could you please direct me where I can find the design doc of elastic learning of PaddlePaddle workload? I can understand a bit of edl
itself from the source code, but I cannot find anything about the elastic PaddlePaddle workload.
Reported by @Haichao-Zhang :
The hyperlinks in this post on Baidu Research Blog to our EDL documents broke.
Could somebody fix it by writing a diff from the current block post content to the right one, and I can ask the administrator of Baidu Research Blog to correct their content.
Thanks!
To my knowledge, edl should rely on the capability of PaddlePaddle workload. In other words, PaddlePaddle must has the ability of elastic learning.
If my understanding is correct, could you please direct me where I can find the design doc of elastic learning of PaddlePaddle workload? I can understand a bit of edl
itself from the source code, but I cannot find anything about the elastic PaddlePaddle workload.
I found that some links in README are 404, I think we should update these links. For example, I am looking for docs for Fault-Tolerant Training in PaddlePaddle. But I cannot find it.
I'd appreciate it if anyone could help me.
Thanks 🥂 🍻
The example shows the min-instance and max-instance value are same for pserver.
pserver:
min-instance: 2
max-instance: 2
resources:
...
Can I set them into different value? Thanks.
The README.md file contains only "How to Build". We also need "How to Run".
To my knowledge, edl should rely on the capability of PaddlePaddle workload. In other words, PaddlePaddle must has the ability of elastic learning.
If my understanding is correct, could you please direct me where I can find the design doc of elastic learning of PaddlePaddle workload? I can understand a bit of edl
itself from the source code, but I cannot find anything about the elastic PaddlePaddle workload.
Need to group scheduling the pods in single operator to prevent resource starving, investigating into kube-arbitrator ATM, have this issue open to keep track with.
Tasks
To my knowledge, edl should rely on the capability of PaddlePaddle workload. In other words, PaddlePaddle must has the ability of elastic learning.
If my understanding is correct, could you please direct me where I can find the design doc of elastic learning of PaddlePaddle workload? I can understand a bit of edl
itself from the source code, but I cannot find anything about the elastic PaddlePaddle workload.
For example, sometimes we need to specify imagePullSecret
or hostNetwork
field for all of the pods, including pservers and trainers.
We intend to submit ReplicaSets and Job for each TrainingJob
this is not implemented yet,
see code from here: https://github.com/PaddlePaddle/edl/blob/develop/pkg/controller.go#L133
To my knowledge, edl should rely on the capability of PaddlePaddle workload. In other words, PaddlePaddle must has the ability of elastic learning.
If my understanding is correct, could you please direct me where I can find the design doc of elastic learning of PaddlePaddle workload? I can understand a bit of edl
itself from the source code, but I cannot find anything about the elastic PaddlePaddle workload.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.