Comments (3)
I also found that there were warnings when loading the model. Is this the reason?
backbone.feature_extractor.,category_head.clip.clip.,clip_head.clip.clip.
WARNING:fvcore.common.checkpoint:Removed 2483 ignored_state_dict keys from missing_keys
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.2.0.shortcut.weight' to the model due to incompatible shapes: (512, 2560, 1, 1) in the checkpoint but (512, 1280, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.2.0.conv1.weight' to the model due to incompatible shapes: (128, 2560, 1, 1) in the checkpoint but (128, 1280, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.3.0.shortcut.weight' to the model due to incompatible shapes: (512, 1920, 1, 1) in the checkpoint but (512, 1280, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.3.0.conv1.weight' to the model due to incompatible shapes: (128, 1920, 1, 1) in the checkpoint but (128, 1280, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.4.0.shortcut.weight' to the model due to incompatible shapes: (512, 960, 1, 1) in the checkpoint but (512, 640, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.4.0.conv1.weight' to the model due to incompatible shapes: (128, 960, 1, 1) in the checkpoint but (128, 640, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.5.0.shortcut.weight' to the model due to incompatible shapes: (512, 640, 1, 1) in the checkpoint but (512, 320, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Skip loading parameter 'backbone.feature_projections.5.0.conv1.weight' to the model due to incompatible shapes: (128, 640, 1, 1) in the checkpoint but (128, 320, 1, 1) in the model! You might want to double check if this is expected.
WARNING:fvcore.common.checkpoint:Some model parameters or buffers are not found in the checkpoint:
backbone.feature_projections.2.0.conv1.weight
backbone.feature_projections.2.0.shortcut.weight
backbone.feature_projections.3.0.conv1.weight
backbone.feature_projections.3.0.shortcut.weight
backbone.feature_projections.4.0.conv1.weight
backbone.feature_projections.4.0.shortcut.weight
backbone.feature_projections.5.0.conv1.weight
backbone.feature_projections.5.0.shortcut.weight
from sd-dino.
Thank you for your interest in our research and for bringing these issues to our attention.
For the get_mask
function, it is designed to return an all-1 matrix when it fails to detect a valid mask for the specified category. From the information you've provided, it seems that the issue might stem from the incorrect loading of the feature projection layers of the ODISE checkpoint.
Could you please try to modify the model-loading code from
model, aug = load_model(diffusion_ver=VER, image_size=SIZE, num_timesteps=TIMESTEP)
to
model, aug = load_model(diffusion_ver=VER, image_size=SIZE, num_timesteps=TIMESTEP, decoder_only=False)
.
This adjustment may better align with the U-Net features utilized by ODISE.
Alternatively, you might want to consider other segmentation methods, such as SAM, for obtaining the object mask. Our choice to use ODISE is mainly for convenience, and the mask is primarily for visualization purposes only.
I hope this could help resolve your queries. Please feel free to reach out If you encounter further issues.
from sd-dino.
Thank you for your thoughtful response and helpful suggestions.
Modifying the model loading code to model, aug = load_model(diffusion_ver=VER, image_size=SIZE, num_timesteps=TIMESTEP, decoder_only=False) did resolve the issue.
from sd-dino.
Related Issues (16)
- Collab Demo HOT 1
- Code for PCA HOT 3
- Object Swapping (with refinement process) HOT 2
- how to fix the issue of 'RuntimeError: Panoptic/odise_label_coco_50e.py not available in Model Zoo!' HOT 7
- cannot `get_mask` when I vary the cuda device HOT 3
- ValueError: a must be greater than 0 unless no samples are taken HOT 5
- AttributeError: module 'keras.backend' has no attribute 'is_tensor' HOT 3
- Details about how to extract sd features HOT 1
- Questions about sd features HOT 2
- Installation issues for Mask Former HOT 2
- Result different from demo_vis_features.ipynb HOT 4
- License? HOT 1
- The image size for extracting the Dinov2-pretrained-model HOT 2
- Establish environment HOT 3
- Model parameter mismatch HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sd-dino.