callsys / genpromp Goto Github PK
View Code? Open in Web Editor NEW[ICCV 2023] Generative Prompt Model for Weakly Supervised Object Localization
License: Apache License 2.0
[ICCV 2023] Generative Prompt Model for Weakly Supervised Object Localization
License: Apache License 2.0
Thanks for your great work! There is an issue during testing.
When using python main.py --function test --config configs/cub_stage2.yml --opt "{'test': {'load_token_path': 'ckpts/cub983/tokens/', 'load_unet_path': 'ckpts/cub983/unet/', 'save_log_path': 'ckpts/cub983/log.txt'}}" for evaluation, I found that self.step_store、self. attention_store and self.attention_maps are all empty. Would you please tell me where is wrong?
Looking forward to your reply!
Thanks for your greate work and sharing.
How do you do the visulization about attention activation?
Thans.
I am currently looking into the implementation details of the model training process, particularly focusing on the model saving mechanism as delineated in the code. In if block, on line 489, it is observed that the model is persistently saved at the conclusion of each training epoch. However, the methodology employed for the selection of the optimal model based on the test/validation set performance remains unclear.
Could you kindly provide an elucidation on the criteria or algorithm used for identifying the most effective epoch based on the validation/test set? This clarification will significantly aid in understanding the overall model selection strategy within the training loop.
Thank you for your assistance.
Hello,
The problem is that the parameter w
or "{'test': {'combine_ratio': 0.6}}"
in the readme.md doesn't seem to work when running inference.
I tried setting values 0, 1, 0.1, 0.9 and compared to the default result of w=0.6
(Also tested in 3 trained instances) ,this 11 result are the same, like :
Cls@1:0.938 Cls@5:0.988 Loc@1:0.481 Loc@5:0.506 Loc_gt:0.514
M-ins:0.000 Part:0.008 More:0.440 Right:0.481 Wrong:0.008 Cls:0.062
(I'm trying to replicate how fr
or fd
works alone.)
I confirm that I successfully adjusted the parameters ( source code location and output.INFO below ),
"INFO: Test Class [0-9]: [dataset: cub] [eval mode: top1] [cam thr: 0.23] [combine ratio: 0.9]"
.
So I don't understand why the result will be the same.(or may be )
After reading the paper, I understood w
as:
fc = fd( 1-w ) + fr ( w )
w>>0,fc>>fd, heatmap'box will small and incomplete;
w>>1,fc>>fr, heatmap'box will much bigger cause deteriorated by background noise;
I successfully trained this project on one 3090 24G
close to the paper,
by reduced train.batch_size
and set larger gradient_accumulation_steps
.
(This parameter is set w=0.5
when train_unet.)
Thank you for the excellent paper and the implemented code. I have a point of confusion. In Figure 3 of the paper, only v_r is colored in orange.
Does this mean that among the embeddings for each word in "a photo of a ", only the word embedding corresponding to is trainable?
However, I find the following part of your code confusing:
# in datasets/base.py
def init_embeddings(self, text_encoder):
token_embeds = text_encoder.get_input_embeddings().weight.data.clone()
for token in self.cat2tokens:
meta_token_id = self.tokenizer.encode(token['meta_token'], add_special_tokens=False)[0]
concept_token_id = self.tokenizer.encode(token['concept_token'], add_special_tokens=False)[0]
token_embeds[concept_token_id] = token_embeds[meta_token_id]
text_encoder.get_input_embeddings().weight = torch.nn.Parameter(token_embeds)
return text_encoder
This code sets token_embeds as trainable by making it a torch.nn.Parameter.
However, contrary to what is shown in Figure 3, this seems to make the entire token_embeds trainable, not just the concept token vector corresponding to .
Could you please clarify my confusion? Thank you very much.
P.S. If my understanding is correct, I believe the following lines of code would be necessary to make only the concept token vector trainable:
text_encoder.get_input_embeddings().weight.requires_grad = False
text_encoder.get_input_embeddings().weight[concept_token_id].requires_grad = True
Thanks for the great work!
I have some questions regarding the two types of embeddings, or tokens, mentioned in the paper.
Prior to the training process, the concept tokens are initialized using the meta tokens.
However, I would like to clarify what happens once the training commences.
Do the meta tokens remain static and not participate in the entire training process? Is it solely the concept tokens that are involved throughout the entire training process?
Thanks for the great work!
Any Colab demo on inference on arbitrary images (not in ImageNet)?
Thank you for your great works.
I want to use your pretrained weights but your given google drive link does not work.
And I also tried to download the Baidu file, but the program is all chinese and I failed to download it.
Can you check the google drive link?
Thank you!
I think your Google Drive Download Link path is invalid.
Please check your README.md
When click the Google Drive Download Link attatched, only reload the page of the repo.
Thank you for your appreciate :)
Hi,
I tried running your code on my custom dataset. Training works well, but I keep encountering a "tuple index out of range" error during the test phase at the following line of code:
representative_embeddings = [text_encoder(ids.to(device))[0] for ids in data["caption_ids_concept_token"][top_idx]]
To ensure that I didn't make any mistakes in my data loading process, I ran the code with the CUB dataset you provided, but I encountered the same error during the test phase.
I noticed in the GitHub README that you suggest saving the models in ckpts/cub during training, but for testing, you load from the ckpts/cub983 path. Could you explain why this discrepancy exists?
Also, do you have any insights into why this error might be occurring?
Thank you
Hello,
When i run
python main.py --function test --config configs/cub_stage2.yml --opt "{'test': {'load_token_path': 'ckpts/cub983/tokens/', 'load_unet_path': 'ckpts/cub983/unet/', 'save_log_path': 'ckpts/cub983/log.txt'}}”
I am encountering this error
Traceback (most recent call last): File "/p/project/atmlaml/benassou1/ega/GenPromp/main.py", line 646, in <module> eval(args.function)(config) File "/p/project/atmlaml/benassou1/ega/GenPromp/main.py", line 300, in test noise_pred = unet(noisy_latents, timesteps, combine_embeddings).sample File "/p/software/juwelsbooster/stages/2023/software/PyTorch/1.12.0-foss-2022a-CUDA-11.7/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/p/project/atmlaml/benassou1/ega/GenPromp/sc_venv_template/venv/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py", line 615, in forward sample = upsample_block( File "/p/software/juwelsbooster/stages/2023/software/PyTorch/1.12.0-foss-2022a-CUDA-11.7/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/p/project/atmlaml/benassou1/ega/GenPromp/sc_venv_template/venv/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py", line 1813, in forward hidden_states = attn( File "/p/software/juwelsbooster/stages/2023/software/PyTorch/1.12.0-foss-2022a-CUDA-11.7/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/p/project/atmlaml/benassou1/ega/GenPromp/sc_venv_template/venv/lib/python3.10/site-packages/diffusers/models/transformer_2d.py", line 265, in forward hidden_states = block( File "/p/software/juwelsbooster/stages/2023/software/PyTorch/1.12.0-foss-2022a-CUDA-11.7/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/p/project/atmlaml/benassou1/ega/GenPromp/sc_venv_template/venv/lib/python3.10/site-packages/diffusers/models/attention.py", line 321, in forward ff_output = self.ff(norm_hidden_states) File "/p/software/juwelsbooster/stages/2023/software/PyTorch/1.12.0-foss-2022a-CUDA-11.7/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/p/project/atmlaml/benassou1/ega/GenPromp/sc_venv_template/venv/lib/python3.10/site-packages/diffusers/models/attention.py", line 379, in forward hidden_states = module(hidden_states) File "/p/software/juwelsbooster/stages/2023/software/PyTorch/1.12.0-foss-2022a-CUDA-11.7/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/p/software/juwelsbooster/stages/2023/software/PyTorch/1.12.0-foss-2022a-CUDA-11.7/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward return F.linear(input, self.weight, self.bias) RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 39.56 GiB total capacity; 7.06 GiB already allocated; 1.94 MiB free; 17.07 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
I changed the batch size to 1, reduced the size of the image, max_split_size_mb, and still does not work. Could you please help me to fix this problem ?
First and foremost, I'd like to express my profound gratitude for the outstanding paper and the code implementation. I have one point of curiosity.
In attempting train_unet
, isn't it the case that the initial weights of each category token pretrained in train_token
are not used?
In the code, train_unet
is executed with split="train"
. Given this, just like when running train_token
, wouldn't the initial weights of the concept_token
in the text_encoder
be initialized identically to the initial weights of the meta_token
?
Since all parameters
of the text_encoder
are frozen during train_unet
, wouldn't this mean that the unet is fine-tuned with the initial weights of both the meta_token
and concept_token
being the same?
In the Loss formula (5) mentioned in the paper, it is depicted as in the linked image. This Loss seems to utilize f* (pretrained initial weight), hence my query.
Thank you always for your hard work.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.