researchmm / generate-it Goto Github PK
View Code? Open in Web Editor NEWA collection of models for image<->text generation in ACM MM 2021.
License: MIT License
A collection of models for image<->text generation in ACM MM 2021.
License: MIT License
In train.py(103-110)
103: # for image
104: _visual_mask = torch.zeros((batch_size, visual_token_num), dtype=torch.float32, device=device)
105: # need to mask token content in selected_idx for prediction/generation
106: num_masks = random.randint(max(1, int(0.1 * visual_token_num)), visual_token_num)
107: selected_idx = random.sample(range(visual_token_num), num_masks)
108: _visual_mask[:, selected_idx] = 1
109: mask_position = (_visual_mask == 1).to(torch.long).view(-1)
110: mask_position = mask_position.nonzero().squeeze()
I think '_visual_mask = 1' means the model can see it, '_visual_mask = 0' is the opposite. The above codes randomly sample mask position, which selects which grid(8*8) the model can see(_visual_mask=1). The position that really needs to be masked is the position where the _visual_mask is equal to 0. So the code on line 109 should be changed to
mask_position = (_visual_mask == 0).to(torch.long).view(-1)
is this right?
Hello, I'm interested in your work!
I want to know what should i do if i want to infer the model by using my own datasets.
Should i build the file like dataset_coco.json, and re-extract cluster and grid features?
Thank you!
@HYPJUDY I am unable to setup the requirements for the project using the "requirements.txt" on my Windows machine as it produces the following error:
I did find a way around this using PyTorch previous versions:
pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html
There is additionally an issue with conflicting tensorboard
versions. I commented out the tensorboard in the "requirements.txt" for now:
Also I believe the yacs
and apex
libraries are required, as they are also not in the requirements and produce errors when running the sample_images.py
file.
Hello! Thank you for opening such nice work!
I am sorry that I am a freshman. Could you tell me how to train the model on my own dataset? Thank you very much!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.