Will you release your real-robot code for data-collection and execution?

Hi,

Is there a code indentation error here?

Hello, congratulations on your excellent work. I am very interested in this work and want to re-implement it. So if you release the source code, please let me know as soon as possible.

Hi,

I noticed that in this code you set the weights to 1 and 0.01. Does this mean that the rendering loss plays a very small role?

In fact, in my task, the total_loss is about 20, while the rendering_loss is very small, such as 1.12.

They themselves differed by an order of magnitude. If multiplied by 0.01, it would be a difference of three orders of magnitude. Is that so?

Alternatively, can you tell me what the optimal ratio is? This saves me some trial and error time. Thanks very much.

Hi,

The role of MLP as I understand is used to decode the representation of encoder3d, function similar to decoder.

If the mlp parameter is too large, will it affect the learning of encoder3d and the robotic arm?

Can an upload of your best model be made available so that it is easier to access? Thanks,

Thank you very much for making the codebase public!

I tried to evaluate public checkpoints following README, and I encountered the error below.

➜  workspace bash scripts/eval.sh GNFACTOR_BC released
[2023-12-27 11:06:50,025][root][INFO] - Using env device cuda:0.
[2023-12-27 11:06:50,073][root][INFO] - Evaluating seed 0.
Traceback (most recent call last):
  File "/root/workspace/GNFactor/eval.py", line 260, in <module>
    main()
  File "/root/.pyenv/versions/3.9.18/lib/python3.9/site-packages/hydra/main.py", line 32, in decorated_main
    _run_hydra(
  File "/root/.pyenv/versions/3.9.18/lib/python3.9/site-packages/hydra/_internal/utils.py", line 346, in _run_hydra
    run_and_report(
  File "/root/.pyenv/versions/3.9.18/lib/python3.9/site-packages/hydra/_internal/utils.py", line 201, in run_and_report
    raise ex
  File "/root/.pyenv/versions/3.9.18/lib/python3.9/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
    return func()
  File "/root/.pyenv/versions/3.9.18/lib/python3.9/site-packages/hydra/_internal/utils.py", line 347, in <lambda>
    lambda: hydra.run(
  File "/root/.pyenv/versions/3.9.18/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 107, in run
    return run_job(
  File "/root/.pyenv/versions/3.9.18/lib/python3.9/site-packages/hydra/core/utils.py", line 128, in run_job
    ret.return_value = task_function(task_cfg)
  File "/root/workspace/GNFactor/eval.py", line 250, in main
    eval_seed(train_cfg,
  File "/root/workspace/GNFactor/eval.py", line 47, in eval_seed
    from agents import gnfactor_bc
  File "/root/workspace/GNFactor/agents/gnfactor_bc/__init__.py", line 1, in <module>
    import agents.gnfactor_bc.launch_utils
  File "/root/workspace/GNFactor/agents/gnfactor_bc/launch_utils.py", line 23, in <module>
    from agents.gnfactor_bc.qattention_gnfactor_bc_agent import QAttentionPerActBCAgent
  File "/root/workspace/GNFactor/agents/gnfactor_bc/qattention_gnfactor_bc_agent.py", line 22, in <module>
    from agents.gnfactor_bc.neural_rendering import NeuralRenderer
  File "/root/workspace/GNFactor/agents/gnfactor_bc/neural_rendering.py", line 11, in <module>
    from agents.gnfactor_bc.models_embed import GeneralizableNeRFEmbedNet
  File "/root/workspace/GNFactor/agents/gnfactor_bc/models_embed.py", line 6, in <module>
    from agents.gnfactor_bc.resnetfc import ResnetFC
  File "/root/workspace/GNFactor/agents/gnfactor_bc/resnetfc.py", line 8, in <module>
    from .attention import Visual3DLangTransformer
  File "/root/workspace/GNFactor/agents/gnfactor_bc/attention.py", line 12, in <module>
    from ldm.modules.diffusionmodules.util import checkpoint
ModuleNotFoundError: No module named 'ldm'

This error is happened here,

I think this line is to load checkpoints using utils of latent-diffusion

Do we also need to install latent-diffusion, which is not instructed in INSTALL.md?

Thanks in advance,

I got an error when starting gnfactor training:
my env setting is below
GPU: nvidia l40s
cuda: 11.7 ( in a docker environment)
ubuntu: 22.04

it seems that xformers is incompatible with torch 2.4.0 but " pip install torchvision --upgrade" in the installation.md occurs torch==2.4.0.

Do you know how to handle this? please help me ! :(

`
Error executing job with overrides: ['method=GNFACTOR_BC', 'rlbench.task_name=GNFACTOR_BC_20240725', 'rlbench.demo_path=/GNFactor/data/train_data', 'replay.path=/GNFactor/replay/GNFACTOR_BC_20240725', 'framework.start_seed=0', 'framework.use_wandb=False', 'method.use_wandb=False', 'framework.wandb_group=GNFACTOR_BC_20240725', 'ddp.num_devices=1', 'ddp.master_port=12345']
Traceback (most recent call last):
File "/GNFactor/GNFactor/train.py", line 100, in main
mp.spawn(run_seed_fn.run_seed,
File "/root/miniconda3/envs/gnfactor/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 282, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")

train_runner.start()

File "/GNFactor/third_party/YARR/yarr/runners/offline_train_runner.py", line 175, in start
loss = self._step(i, batch)
File "/GNFactor/third_party/YARR/yarr/runners/offline_train_runner.py", line 97, in _step
update_dict = self._agent.update(i, sampled_batch)
File "/GNFactor/GNFactor/helpers/preprocess_agent.py", line 58, in update
return self._pose_agent.update(step, replay_sample)
File "/GNFactor/GNFactor/agents/gnfactor_bc/qattention_stack_agent.py", line 47, in update
update_dict = qa.update(step, replay_sample)
File "/GNFactor/GNFactor/agents/gnfactor_bc/qattention_gnfactor_bc_agent.py", line 772, in update
rendering_loss_dict = self._q(obs,
File "/root/miniconda3/envs/gnfactor/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/gnfactor/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/GNFactor/GNFactor/agents/gnfactor_bc/qattention_gnfactor_bc_agent.py", line 227, in forward
rendering_loss_dict = self._neural_renderer(voxel_feat=voxel_grid_feature,
File "/root/miniconda3/envs/gnfactor/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/miniconda3/envs/gnfactor/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/gnfactor/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1636, in forward
else self._run_dd
out = xformers.ops.memory_efficient_attention(q, k, v, attn_bias=None, op=self.attention_op)
File "/root/miniconda3/envs/gnfactor/lib/python3.9/site-packages/xformers/ops/fmha/init.py", line 196, in memory_efficient_attention
return _memory_efficient_attention(
File "/root/miniconda3/envs/gnfactor/lib/python3.9/site-packages/xformers/ops/fmha/init.py", line 294, in _memory_efficient_attention
return _memory_efficient_attention_forward(
File "/root/miniconda3/envs/gnfactor/lib/python3.9/site-packages/xformers/ops/fmha/init.py", line 310, in _memory_efficient_attention_forward
op = _dispatch_fw(inp)
File "/root/miniconda3/envs/gnfactor/lib/python3.9/site-packages/xformers/ops/fmha/dispatch.py", line 98, in _dispatch_fw
return _run_priority_list(
File "/root/miniconda3/envs/gnfactor/lib/python3.9/site-packages/xformers/ops/fmha/dispatch.py", line 73, in _run_priority_list
raise NotImplementedError(msg)
NotImplementedError: No operator found for memory_efficient_attention_forward with inputs:
query : shape=(1, 4096, 1, 512) (torch.float32)
key : shape=(1, 4096, 1, 512) (torch.float32)
value : shape=(1, 4096, 1, 512) (torch.float32)
attn_bias : <class 'NoneType'>
p : 0.0
cutlassF is not supported because:
xFormers wasn't build with CUDA support
Operator wasn't built - see python -m xformers.info for more info
flshattF is not supported because:
xFormers wasn't build with CUDA support
dtype=torch.float32 (supported: {torch.bfloat16, torch.float16})
max(query.shape[-1] != value.shape[-1]) > 128
Operator wasn't built - see python -m xformers.info for more info
tritonflashattF is not supported because:
xFormers wasn't build with CUDA support
dtype=torch.float32 (supported: {torch.bfloat16, torch.float16})
max(query.shape[-1] != value.shape[-1]) > 128
requires A100 GPU
smallkF is not supported because:
xFormers wasn't build with CUDA support
max(query.shape[-1] != value.shape[-1]) > 32
Operator wasn't built - see python -m xformers.info for more info
unsupported embed per head: 512

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
[2024-07-25 07:46:05,821][root][INFO] - Using env device cuda:0.
[2024-07-25 07:46:05,836][root][INFO] - Evaluating seed 0.
[2024-07-25 07:46:06,043][xformers][WARNING] - WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.0.0+cu118 with CUDA 1108 (you have 2.4.0+cu121)
Python 3.9.16 (you have 3.9.19)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
/root/miniconda3/envs/gnfactor/lib/python3.9/site-packages/xformers/triton/softmax.py:30: FutureWarning: torch.cuda.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='cuda') instead.
@custom_fwd(cast_inputs=torch.float16 if _triton_softmax_fp16_enabled else None)
/root/miniconda3/envs/gnfactor/lib/python3.9/site-packages/xformers/triton/softmax.py:87: FutureWarning: torch.cuda.amp.custom_bwd(args...) is deprecated. Pl File "/romaking attention of type 'vanilla-xformers' with 512 in_channels [89/1941]
building MemoryEfficientAttnBlock with 512 in_channels...
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla-xformers' with 512 in_channels
building MemoryEfficientAttnBlock with 512 in_channels...
INFO:fvcore.common.checkpoint:[Checkpointer] Loading from sd://v1-3 ...
INFO:iopath.common.file_io:URL https://huggingface.co/CompVis/stable-diffusion-v-1-3-original/resolve/main/sd-v1-3.ckpt cached in /root/.torch/iopath_cache/CompVis/stable
-diffusion-v-1-3-original/resolve/main/sd-v1-3.ckpt
WARNING:fvcore.common.checkpoint:The checkpoint state_dict contains keys that are not used by the model:
model_ema.{decay, num_updates}
diffusion feature dims: [512, 512, 2560, 1920, 960, 640, 512, 512]
[NeuralRenderer] foundation model diffusion is build. loss weight: 0.01
[NeuralRenderer] rgb loss weight: 1.0
[NeuralRenderer]: True
INFO:root:# Q Params: 44 M
[rank0]:[W725 07:45:58.101337864 reducer.cpp:1400] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the fo
rward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unu
sed parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations
to have unused parameters. (function operator())
Error executing job with overrides: ['method=GNFACTOR_BC', 'rlbench.task_name=GNFACTOR_BC_20240725', 'rlbench.demo_path=/GNFactor/data/train_data', 'replay.path=/GNFactor
/replay/GNFACTOR_BC_20240725', 'framework.start_seed=0', 'framework.use_wandb=False', 'method.use_wandb=False', 'framework.wandb_group=GNFACTOR_BC_20240725', 'ddp.num_dev
ices=1', 'ddp.master_port=12345']
Traceback (most recent call last):
File "/GNFactor/GNFactor/train.py", line 100, in main`

Hello, I used your code, but in the image display, I found that it did not achieve the effect presented in your paper. Instead, I had terrible results.

Can you tell me where the problem is?

The picture below is the result after training for 70k steps

when I run command: bash scripts/gen_demonstrations.sh close_jar, Some problems appear.

File "/home/fy/Downloads/GNFactor/third_party/PyRep/pyrep/backend/sim.py", line 94, in simGetObjectHandle
raise RuntimeError('Handle %s does not exist.' % objectName)
RuntimeError: Handle Panda does not exist.

It may be a problem from PyRep, but I don't know how to solve it.