[Paper] [Project Page]

Authors: Hongjie Fang, Hao-Shu Fang, Sheng Xu, Cewu Lu.

Welcome to the official repository for the TransCG paper. This repository includes the dataset and the proposed Depth Filler Net (DFNet) models.

TransCG dataset is now available on official page. TransCG dataset is the first large-scale real-world dataset for transparent object depth completion and grasping. In total, our dataset contains 57,715 RGB-D images of 51 transparent objects and many opaque objects captured from different perspectives of 130 scenes under various real-world settings. The 3D mesh model of the transparent objects are also provided in our dataset.

Daily Transparent Objects in Dataset	Real-time Tracking System	Robot Collection

The code has been tested under

• Ubuntu 18.04 + NVIDIA GeForce RTX 3090 (CUDA 11.1)
• PyTorch 1.9.0

System dependencies can be installed by:

sudo apt-get install libhdf5-10 libhdf5-serial-dev libhdf5-dev libhdf5-cpp-11
sudo apt install libopenexr-dev zlib1g-dev openexr

Other dependencies can be installed by

pip install -r requirements.txt

Our pretrained checkpoint is available on Google Drive or Baidu Netdisk (Code: c01g). The checkpoint is trained with the default configuration in the configs folder. You can use our released checkpoints for inference or testing. Refer to assets/docs/DFNet.md for details about the depth completion network.

To verify the depth completion results in robotic manipulation, we select the fundamental object grasping as the downstream task of our network. Here are the grasping demos. Refer to assets/docs/grasping.md for details about the grasping method.

You need to create a configuration file for training, testing and inference. See assets/docs/configuration.md for details.

• TransCG (recommended): See TransCG Dataset section;
• ClearGrasp (syn and real): See ClearGrasp official page;
• Omniverse Object Dataset: See implicit-depth official repository;
• Transparent Object Dataset: See KeyPose official page.

For inference stage, there is a Inferencer class in inference.py, you can directly call it for inference.

Example. Given an H x W x 3 RGB image rgb, and an H x W depth image depth (after scaling according to camera parameters), you can use the following code to get the refined depth according to our models.

from inferencer import Inferencer
# Initialize the inferencer. It is recommended to intiailize before starting your task for real-time performance.
inferencer = Inferencer(cfg_file = 'configs/inference.yaml') # Specify your configuration file here.
# Call inferencer for refined depth
refine_depth = inferencer.inference(rgb, depth)

For full code sample, refer to sample_inference.py.

For training from scrach, you need to create a configuration file following instruction of configuration section. Then, execute the following commands to train your own model.

python train.py --cfg [Configuration File]

If you want to fine-tune your model from some checkpoints, you may need to provide resume_lr in configuration file. See assets/docs/configuration.md for details.

For model testing, you also need to create a configuration file following instruction of configuration section. Then, execute the following commands to test the model.

python test.py --cfg [Configuration File]

Note. For testing stage, the checkpoint specified in the configuration file should exist.

@article{fang2022transcg,
    title   = {TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and Grasping},
    author  = {Fang, Hongjie and Fang, Hao-Shu and Xu, Sheng and Lu, Cewu},
    journal = {arXiv preprint arXiv:2202.08471}
    year    = {2022}
}