baaivision / uni3d Goto Github PK

1.How to perform cross-modal retrieval?
2.Do you need to download the complete data set?
3.How long does it take?
I am a beginner and asked some basic questions. I would be very grateful if you could answer me。

thanks for your work, and I have a question about open_clip.create_model_and_transforms.
pretorianed = path/to/clip_model.bin
model, _, preprocess = open_clip.create_model_and_transforms('EVA02-E-14-plus',pretrained=pretrained)
but there are something wrong with that. it takes a few minutes to create model, and I found the memory of this model(I don's create other model) is 20GB. and I change pretrained = None, it can also create the model for a long time , and the memory is also 20GB.
this bug makes me fuses a long time, if it's ok, I need your help. thanks.

Hello to the authors,

First off, thanks so much for sharing your project with the community! It’s been really interesting to dive into. I've noticed that in the zero-shot classification, RGB data is used as input for the Objarverse dataset, but not for Modelnet40 and ScanObjectNN.

Just wondering, is there a particular reason why RGB data isn't part of the setup for Modelnet40 and ScanObjectNN? Any insights you could provide would be super helpful for understanding the approach better and might even impact how I tackle similar issues in my work.

Thanks in advance for any thoughts you might share, and looking forward to hearing from you.

This is an Interesting work!
I'm new to this type of representation learning work, and I have a few inquiries regarding the downstream tasks. I apologize for any inconvenience my questions may cause.

1. How are the downstream tasks conducted? I comprehend that the 3D model is encoded into embeddings, which exist in the same representation space as text or images. Is this a presupposed prior knowledge that you expect all readers to possess?
2. Do you have any quantified baseline for the text-3D retrieval task?

Thanks for your nice work!
Your contribution of designing billion-scale 3D representation models can be seen in Table5 in the experimental section of the paper. But I only found the Uni3d-g model with the best performance in the repo. I'm here to ask for your releasing of other scales of representation model checkpoints, e.g. Uni3d-L and Uni3d-B, so that we could ultilize all these models for further research!
I appreciate your quick response.

Thank you for your very interesting work! How should One-shot Part Segmentation and Point Cloud Painting be used?

It is a good job. I have a question about the Evaluation datasets objaverse-lvis. I have download the test_datasets.zip. However, it seems that its dim is 1280, not 1024. So I can not use it for zero-shot classification directly. Could you provide the right Evaluation datasets objaverse-lvis

Dear Author,

Many thanks for the great work and the nice open-source.

During trying your code, I noticed that the embeding diomension of the point cloud encoder is set to be 1024, which is to match the dimension of the EVA-CLIP. However, I would like to try for some smaller versions of CLIP(as also shown in your paper) which only supports 512 dimention, is it possible to provide the pretrained Uni3D with 512 embedding dimension?

Thanks for the reply in advance.

Is it the same text generation method as openshape?

Dear authors,

Thank you for your interesting work and your extensive experiments. 🎉
When will the code be released for the segmentation and the painting task? 🎨

Kind regards
Jarne

This is a very meaningful job!

1. I would like to ask if all training samples are input with color during the pre-training process?
2. In the zero-shot classification experiment, for ModelNet40, color input was not used. Should these dimensions be deleted or replaced with a fixed value (0.4)?
3. If color is not used as input in pre-training, will there be any impact on downstream tasks?

When I use this pointnet2_ops_lib for pointnet2, I found out the atomicAdd() function in the .cu file is not fixed in order when called, resulting in a numerical error in the sum of floating points as a result of gradient calculations. And set_seed() in python is useless.
I found that your amazing network is based on this package, and how do you deal with this non-determinism?
If determinism is not guaranteed, ablation experiments can be cumbersome.
Thank you!

Thank you for open-sourcing this work!
Could you please add licensing?

Thanks for sharing the paper and code. It's a great work.

Since Uni3D has scaled the point encoder to 1B parameters, that are rather big. But the scale of Objaverse 1.0 is only 800K 3D objects, I think it is still relatively small to support the 1B-scale point encoder pre-training and the generalization is still far behind the counterparts in image and text field.

Now Objaverse-xl is released, which contains 10M+ 3D objects, does your team have such a plan to pre-train on larger Objaverse-xl? I think BAAI has the computing source to finish such a task. How do you think about it?

Dear Authors,

Thanks for the great work!

I noticed that the templates for the zeroshot classification to be a list of prompts(in the "templates.jason" file), may I ask if these templates come from some previous works? Or why do you choose such templates?

Thank you for your reply in advance.

Best regards

Thanks for sharing the paper, code and weights. It's a great work!

One point confuses me is that in the model zoo, you have two uni3d_g models pre-trained with the Ensemble datasets, as the red box highlights below.

But their zero-shot accuracy is quite different, uni3d_g with the fire symbol is much better than uni3d_g, so what's the difference between these two models?

Looking forward to your reply. Thanks.

I am trying to use the command line- pip install "git+git://github.com/erikwijmans/Pointnet2_PyTorch.git#egg=pointnet2_ops&subdirectory=pointnet2_ops_lib"

But it is showing error like-

Collecting pointnet2_ops
Cloning git://github.com/erikwijmans/Pointnet2_PyTorch.git to /tmp/pip-install-yeknjq64/pointnet2-ops_0f5c19c3381948d0b75fd75f60b29bcb
Running command git clone --filter=blob:none --quiet git://github.com/erikwijmans/Pointnet2_PyTorch.git /tmp/pip-install-yeknjq64/pointnet2-ops_0f5c19c3381948d0b75fd75f60b29bcb
fatal: unable to connect to github.com:
github.com[0: 20.205.243.166]: errno=Connection timed out

error: subprocess-exited-with-error

× git clone --filter=blob:none --quiet git://github.com/erikwijmans/Pointnet2_PyTorch.git /tmp/pip-install-yeknjq64/pointnet2-ops_0f5c19c3381948d0b75fd75f60b29bcb did not run successfully.
│ exit code: 128
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git clone --filter=blob:none --quiet git://github.com/erikwijmans/Pointnet2_PyTorch.git /tmp/pip-install-yeknjq64/pointnet2-ops_0f5c19c3381948d0b75fd75f60b29bcb did not run successfully.
│ exit code: 128
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Hello, I'm using Uni3D to try to extract embeddings from 3D models from Objaverse. The format of 3d models are .glb, and I don't know how to transform it into .npy that you use in Objaverse-LVIS. Do you know how to make it or where can I find it?

BTW, the code snippet below is how I manage to extract the embedding of 3D models. Am I missing something?

@torch.no_grad()
def encode(dataloader):
    model = create_uni3d()
    model.eval()
    model.to(device)
    embeddings = []
    for (pc, _, _, rgb) in tqdm(dataloader, desc="Extracting embeddings"):
        pc = pc.to(device=device, non_blocking=True)
        rgb = rgb.to(device=device, non_blocking=True)
        feature = torch.cat((pc, rgb), dim=-1)
        pc_features = model.encode_pc(feature)
        pc_features = pc_features / pc_features.norm(dim=-1, keepdim=True)
        embeddings.append(pc_features)
    embeddings = torch.cat(embeddings, dim=0)
    return embeddings

Thank you for your excellent work. I have a question about applying a point cloud classification model trained on object datasets to 3D scene segmentation tasks like ScanNet. Does the model require fine-tuning to adapt to the complexities of full scene segmentation, or can the original model be directly applied to this task? Your insights would be very helpful. Thank you!

Thank you for your interesting work! I just wonder how do you perform open-world understanding on scannet dataset?

Thank you for your great work. I am a newbie in this area and I would like to know what kind of GPU device is required when evaluating the model? I have two 3090 NVIDIA graphics cards, but this doesn't seem to be enough for evaluating the model. Is there any suitable method? Thanks! !