Comments (4)
grwlf
commented on September 4, 2024
Note: printing cdfs before line 603 shows
[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 0.0, 0.0], [0.0, 11010048.0, 151617536.0], [0.0, 22544384.0, 303235072.0], [0.0, 45613056.0, 370409472.0], [0.0, 68681728.0, 370409472.0], [0.0, 80740352.0, 420773888.0], [0.0, 81002496.0, 420773888.0], [0.0, 92536832.0, 420773888.0], [0.0, 104071168.0, 420773888.0], [0.0, 116129792.0, 454361088.0], [0.0, 116391936.0, 454361088.0], [0.0, 127926272.0, 454361088.0], [0.0, 139460608.0, 454361088.0], [0.0, 150994944.0, 454361088.0], [0.0, 163053568.0, 487948288.0], [0.0, 163315712.0, 487948288.0], [0.0, 174850048.0, 487948288.0], [0.0, 186384384.0, 487948288.0], [0.0, 209401856.0, 529928192.0], [0.0, 220411904.0, 529928192.0], [0.0, 220674048.0, 529928192.0], [0.0, 220936192.0, 529928192.0], [0.0, 231946240.0, 529928192.0], [0.0, 242956288.0, 529928192.0], [0.0, 253966336.0, 529928192.0], [0.0, 265500672.0, 580292608.0], [0.0, 265762816.0, 580292608.0], [0.0, 276772864.0, 580292608.0], [0.0, 287782912.0, 580292608.0], [0.0, 298792960.0, 580292608.0], [0.0, 310327296.0, 630657024.0], [0.0, 310589440.0, 630657024.0], [0.0, 321599488.0, 630657024.0], [0.0, 332609536.0, 630657024.0], [0.0, 343619584.0, 630657024.0], [0.0, 354629632.0, 630657024.0], [0.0, 366163968.0, 681021440.0], [0.0, 366426112.0, 681021440.0], [0.0, 377436160.0, 681021440.0], [0.0, 388446208.0, 681021440.0], [0.0, 786442240.0, 832787040.0]]
from pipedream.
deepakn94
commented on September 4, 2024
To solve this particular error, you can just comment out the render_bar_graphs_and_cdfs() method call...I am worried that the root cause is something else though -- seems like the times for each operator are all zeros?
from pipedream.
grwlf
commented on September 4, 2024
I did some debugging and my guess is that assigns at line 324 are bypassed by the above 'continue' branches. summary_elem could already contain _time fields due to zero-initialization at the beginning of the training. Could you please check?
from pipedream.
heguangxin
commented on September 4, 2024
I did some debugging and my guess is that assigns at line 324 are bypassed by the above 'continue' branches.
summary_elemcould already contain_timefields due to zero-initialization at the beginning of the training. Could you please check?
i got the same problem: times for each operator are all zeros. Did you figure it out?
from pipedream.
Related Issues (20)
- What's the latest version of PyTorch supported? HOT 10
- What is the meaning of `antichain` in optimizer_graph_hierarchical.py ?
- Can the profiler handle dynamic graphs?
- Hanging with [4,3,1] GPU assignment
- Communication error when training with Pipedream HOT 6
- Planner for PipeDream-2BW HOT 5
- Running a transformer module
- Handling uneven number of batches per replicated instance of a layer
- GPU Peer2Peer communication via --num_ranks_in_server argument HOT 1
- Resource temporarily unavailable
- To run PipeDream_2BW branch without --recompute_step
- The BLEU score of translation model seems abnormal. The model doesn't seem to train effectively.
- GPT2 355m model convergence with 2BW training
- Is there AllReduce in data parallelism? HOT 6
- How is the Double-Buffered Weight Mechanism implemented?
- Supporting T5
- The arguments of self.start_helper_thread() should be more flexible instead of fixed as int64.
- Question about time complexity of PipeDream-2BW's planner algorithm
- Question about PipeDream's optimizer
- AttributeError: module 'models.resnet50.resnet50' has no attribute 'model' HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pipedream.