Comments (9)
thuanz123
commented on August 23, 2024
What is the size of your dataset ? If I remember correctly, it takes like 10 epochs of training on imagenet for the output to be good. So that is about 12M iterations
from enhancing-transformers.
thuanz123
commented on August 23, 2024
Oh sorry I mis-calculate a bit. 10 epochs on ImageNet with batch size of 128 is about 100K iterations. But the dataset size is also matters, it does not work well with small dataset
from enhancing-transformers.
commented on August 23, 2024
Dataset is FFHQ, during the 12000 steps, the outputs are very strages, so
i am worried about the model is wrong.
From: "Thuan H. ***@***.***>
Date: Sun, Jun 25, 2023, 13:05
Subject: [External] Re: [thuanz123/enhancing-transformers] stage2 transform
(Issue #16)
To: "thuanz123/enhancing-transformers"<
***@***.***>
Cc: ***@***.***>, "Author"<
***@***.***>
What is the size of your dataset ? If I remember correctly, it takes like
10 epochs of training on imagenet for the output to be good. So that is
about 12M iterations
—
Reply to this email directly, view it on GitHub
<#16 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A7ZMO7CTI2H56JQKXCKZNI3XM7BKNANCNFSM6AAAAAAZS5KQ4A>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
from enhancing-transformers.
thuanz123
commented on August 23, 2024
Actually, only the stage1 code is good. The code for stage2 is not good. Most of the time, it produces bad results but sometimes good so I recommend using another code until I can find what wrong
from enhancing-transformers.
commented on August 23, 2024
Thank you. When dataset is small, it can not generate good images? And I
take conds as "
conds = torch.zeros(images.shape[0], 1, device=device, dtype=torch.long)"
for unconditional generation, shape is (b, 1). Do you think it is correct?
From: "Thuan H. ***@***.***>
Date: Sun, Jun 25, 2023, 13:09
Subject: [External] Re: [thuanz123/enhancing-transformers] stage2 transform
(Issue #16)
To: "thuanz123/enhancing-transformers"<
***@***.***>
Cc: ***@***.***>, "Author"<
***@***.***>
Oh sorry I mis-calculate a bit. 10 epochs on ImageNet with batch size of
128 is about 100K iterations. But the dataset size is also matters, it does
not work well with small dataset
—
Reply to this email directly, view it on GitHub
<#16 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A7ZMO7G5GI5K2SBLIGB7K4DXM7BW5ANCNFSM6AAAAAAZS5KQ4A>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
from enhancing-transformers.
commented on August 23, 2024
Do you have any suggestions? Maybe VQGAN or RA-Transformer official codes?
From: "Thuan H. ***@***.***>
Date: Sun, Jun 25, 2023, 13:12
Subject: [External] Re: [thuanz123/enhancing-transformers] stage2 transform
(Issue #16)
To: "thuanz123/enhancing-transformers"<
***@***.***>
Cc: ***@***.***>, "Author"<
***@***.***>
Actually, only the stage1 code is good. The code for stage2 is not good.
Most of the time, it produces bad results but sometimes good so I recommend
using another code until I can find what wrong
—
Reply to this email directly, view it on GitHub
<#16 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A7ZMO7CDLMTONPT2GGJ6GFLXM7CELANCNFSM6AAAAAAZS5KQ4A>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
from enhancing-transformers.
commented on August 23, 2024
Thank your reply. Another question, what numbers of your gpus? 8 A100-80G?
The experiments take long time. Can you reproduce the metrics same as
papers for stage 1 and stage2.
From: "Thuan H. ***@***.***>
Date: Sun, Jun 25, 2023, 13:12
Subject: [External] Re: [thuanz123/enhancing-transformers] stage2 transform
(Issue #16)
To: "thuanz123/enhancing-transformers"<
***@***.***>
Cc: ***@***.***>, "Author"<
***@***.***>
Actually, only the stage1 code is good. The code for stage2 is not good.
Most of the time, it produces bad results but sometimes good so I recommend
using another code until I can find what wrong
—
Reply to this email directly, view it on GitHub
<#16 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A7ZMO7CDLMTONPT2GGJ6GFLXM7CELANCNFSM6AAAAAAZS5KQ4A>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
from enhancing-transformers.
thuanz123
commented on August 23, 2024
- Yeah you can do simple thing like zero latent code for unconditional
- If you want to re-use the provided pretrained weight then you must use VQ-GAN code, if you want to use RQ-VAE code then you must train a compatible stage1 model
- Training ViT-VQGAN takes a really long time. In the paper they use like 128 TPUs which I certainly do not have access to so I cannot match their numbers unfortunately
from enhancing-transformers.
commented on August 23, 2024
Thanks for your reply. It is very helpful for me. Yeah, I think the reason
that ViT VQGAN takes a long time is transformer architecture, it really
needs a lot of time.
From: "Thuan H. ***@***.***>
Date: Sun, Jun 25, 2023, 13:23
Subject: [External] Re: [thuanz123/enhancing-transformers] stage2 transform
(Issue #16)
To: "thuanz123/enhancing-transformers"<
***@***.***>
Cc: ***@***.***>, "Author"<
***@***.***>
1. Yeah you can do simple thing like zero latent code for unconditional
2. If you want to re-use the provided pretrained weight then you must
use VQ-GAN code, if you want to use RQ-VAE code then you must train a
compatible stage1 model
3. Training ViT-VQGAN takes a really long time. In the paper they use
like 128 TPUS which I certainly do not have access to
—
Reply to this email directly, view it on GitHub
<#16 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A7ZMO7COBD5VWEZFOSWCZX3XM7DL3ANCNFSM6AAAAAAZS5KQ4A>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
from enhancing-transformers.
Related Issues (20)
- Pre-norm and Post-norm HOT 6
- An inplace operation in the forward process HOT 1
- Reconstruction Visualization HOT 2
- stage2 transformer HOT 2
- stage1 pretraining HOT 7
- Model license HOT 4
- about training loss HOT 1
- The question is done. HOT 1
- ImageNet version.
- Reconstruction results HOT 4
- Learning rate and scheduler for stage 1 training. HOT 1
- why not used pretrained discriminator
- Incomplete implementation of RQ-VAE
- raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['where', 'cl']' returned non-zero exit status 1. HOT 5
- Results HOT 5
- Smaller images HOT 10
- OOM for imagenet_gpt_vitvq_base and a 100M params GPT on A100 40G HOT 3
- Pretrained Stage 2 Transformer for ViT-VQGAN HOT 8
- Training time and number of GPUS HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from enhancing-transformers.