Comments (17)
See #6. Thanks!
from meter.
@zdou0830 Thanks!
I have another question about the codes.
I tried your pretrained model to finetune VQA v2 with default setting.
However, the score of val is only around 72.55.
It should be more than 80..
Could you share your experiment settings of finetuning VQA v2 and pretraining tasks?
from meter.
The command should be
python run.py with data_root=$DATA_DIR num_gpus=8 num_nodes=1 task_finetune_vqa_clip_bert per_gpu_batchsize=4 clip16 text_roberta image_size=576 clip_randaug load_path=meter_clip16_288_roberta_pretrain.ckpt
Hope this thread would help #7.
from meter.
This setting is the same as mine ...
from meter.
If I use your finetuned model to do the test only task of VQA v2, the result is 77.66.
This model works.
from meter.
from meter.
You can try testing the last checkpoint and submitting the resulting json file to evalai.
from meter.
The result from evalai is 71.53.
Should I finetune the model with more step?
from meter.
The VQA dataset can be downloaded here: https://drive.google.com/file/d/1qT7YWHpLg-fAL43daKlOsYx2EbbQk--d/view?usp=sharing.
The training command is
python run.py with data_root=$DATA_DIR num_gpus=8 num_nodes=1 task_finetune_vqa_clip_bert per_gpu_batchsize=4 clip16 text_roberta image_size=576 clip_randaug load_path=meter_clip16_288_roberta_pretrain.ckpt
The testing command is
python run.py with data_root=$DATA_DIR num_gpus=8 num_nodes=1 test_only=True task_finetune_vqa_clip_bert per_gpu_batchsize=4 clip16 text_roberta image_size=576 load_path=last.ckpt
The provided VQA-finetuned checkpoint is trained in this way, so if you follow these steps correctly, you should be able to get a score of ~77.6 on test-dev. I didn't look at the dev scores and the number of training epochs was set to 10 as in config.py. The fine-tuning took about 2 days on 8 V100s for reference.
from meter.
Ok, I will have a try! Thank you for your patient.
from meter.
I found a problem.
If I use only one node to finetune the pretrained model, the result will be better than two nodes (around 5% in VQA v2 Val).
That might be the reason.
May you share your pretraining log?
I will pretrain the model in two nodes.
I want to know the different between one node and two nodes.
from meter.
I didn't save the logs, but I did pre-train the models with 1/2/4 nodes and there were no significant differences, so I'd suggest you to debug your multi-node training settings.
from meter.
I also met the same problem, which probably results from multi-node training settings. I use slurm to multi-node train. May you share your train bash file, if you also use slurm.
from meter.
I didn't use slurm, but I uploaded the running file for distributed training on Microsoft machines (https://github.com/zdou0830/METER/blob/main/azure_distributed_run.py). Not sure if this is helpful.
from meter.
ok, thank u~
from meter.
I also met the same problem, which probably results from multi-node training settings. I use slurm to multi-node train. May you share your train bash file, if you also use slurm.
@jiyt17 Hello, have you solved the problems you encountered before?
from meter.
@zdou0830 Thanks!
I have another question about the codes.
I tried your pretrained model to finetune VQA v2 with default setting. However, the score of val is only around 72.55.
It should be more than 80..
Could you share your experiment settings of finetuning VQA v2 and pretraining tasks?
@wanng-ide Hello, have you solved the problems you encountered before?
from meter.
Related Issues (20)
- About license HOT 2
- How much is the per gpu batch size? HOT 5
- The results of COCO IR/TR HOT 3
- question about the pre-trained weights HOT 2
- Pre-trained models for the Merged Attention Model? HOT 2
- ValueError and AttributeError HOT 4
- questions about vqa HOT 2
- Why the test results are different using same data? HOT 1
- The model meter_clip16_288_roberta_flickr.ckpt is inconsistent with the network weight parameter dimension HOT 4
- The last checkpoint or the best one on the Val split? HOT 3
- The training set of using different pretraining datasets. HOT 2
- Pretrained weights of CLIP-ViT-224/32
- Inference with Fine-tuned SNLI Model HOT 4
- Unable to train models faster with more gpus HOT 3
- pretraining task HOT 4
- Image Caption Script HOT 5
- > Hello, you may check 'Evaluation on Downstream Tasks' in README.md and find whether it's what you want? ![image](https://user-images.githubusercontent.com/71176040/201471719-fc32a022-201d-4c9d-acee-03d39451da56.png)
- Is P100 okay? HOT 1
- training steps HOT 1
- Resume pre-training using "resume_from_checkpoint" in pytorch lightning HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from meter.