Comments (2)
Hi,
The resnet example is also a convnet example. Let us know if that works for you.
from mup.
Feel free to open this back up if the example doesn't work for you.
from mup.
Related Issues (20)
- Positional Embeddings should be MuReadout parameters ? HOT 2
- Warmup schedule when changing the number of tokens/steps (GPT-3 experiment detail)
- Reproducing the training loss vs learning rates curve on MLP HOT 5
- Once the best HPs have been found, does the final model have to be trained with `mup` or can one just use the found HPs and train the model in a standard way?
- Is it possible to also scale the depth of the model? HOT 5
- _rescale_parameters() inconsistent with the paper for the tied embedding scenario? HOT 2
- µTransfer across batch size && weight decay setting
- Some questions about the implementation of muP.
- Interpreting jitter in coordcheck HOT 2
- FSDP support? HOT 3
- Usage with torch.compile in Pytorch 2? HOT 2
- dim_feedforward
- Unclear `assert_hidden_size_inf` triggers HOT 1
- About Learning rate decay HOT 2
- Questions for training gpt-2 using mup HOT 6
- Reproducing the validation accuracy vs learning rates curve on ResNet HOT 1
- coord_check for model that returns loss function directly
- Reproducing Figure 1 using 'examples/Transformer/main.py'
- mu parametrization for gated-mlp and group-query attention
- Increasing coord check for the network output
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mup.