Comments (2)
ghostplant
commented on May 16, 2024
This project is not bonded to DeepSpeed, so it is also compatible for other language models and frameworks not depending on DeepSpeed (e.g. SWIN, Fairseq, etc).
In the meanwhile, DeepSpeed's Top-1 gating can be also boosted if you have Tutel project installed in your environment: https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py#L46
But that would just benefit from part of Tutel's kernel optimizations and new features since Tutel >= 0.2.x would not be leveraged.
from tutel.
Hap-Zhang
commented on May 16, 2024
Ok, I see. Thank you very much for your reply.
from tutel.
Related Issues (20)
- Multi-nodes training is much more slower than single node HOT 1
- [installation errors] fatal error: nccl.h: No such file or directory HOT 1
- RuntimeError: No such operator tutel_ops::cumsum HOT 10
- How the experts' gradients are handled under data parallelism? HOT 1
- All2All precision always in fp32 HOT 1
- tutel/jit_kernels/sparse.py torch.float16 There is a bug in the calculation: the cuda calculation result is inconsistent with the CPU calculation result and the array is out of bounds HOT 1
- [Bug]The function func_fwd is calculated inconsistent on the cpu and gpu HOT 1
- ImportError: cannot import name 'tutel_custom_kernel' from 'tutel.impls.jit_compiler' HOT 12
- about compute_location and locations HOT 1
- INTERNAL ASSERT FAILED HOT 5
- Training with Data and Expert Parallelism HOT 5
- Can this package support the one-gpu machine HOT 5
- how to use tutel on Megatron Deepspeed HOT 4
- numpy not in requirements HOT 5
- tutel is slower than the naive p2p using 2DH for small scale HOT 3
- RuntimeError: (0) == (cuModuleLoadDataEx(&hMod, image.c_str(), sizeof(options) / sizeof(*options), options, values)) INTERNAL ASSERT FAILED HOT 3
- Non-surface function utilities only work for contiguous input data HOT 12
- How to implement Fairseq-MoE training checkpoint like Swin-MoE? HOT 1
- [Question] Why use datatype ncclInt8 in nccl_all_to_all_scatter_async.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tutel.