Thomas Lips,
Victor-Louis De Gusseme,
Francis wyffels
AI and Robotics Lab (AIRO) ,
Ghent University
2nd workshop on Representing and Manipulating Deformable Objects (RMDO)
IEEE International Conference on Robotics and Automation (ICRA), 2022
Paper | Poster | Spotlight Talk | Technical Video
In this work we learn to detect the corners of cloth items. We use procedural data generation to train a convolutional neural network from synthetic data, enabling low-cost data collection. We evaluate the zero-shot performance of this network on a unimanual robot setup for folding towels and find that the grasp and fold success rates are 77% and 53%, respectively. We conclude that learning keypoint detectors from synthetic data for such tasks is a promising research direction, discuss some failures and relate them to future work.
The codebase has 3 main components:
We also provide the pretrained weights and all training logs.
To generate synthetic images of unfolded cloths, we make use of Blender, version 3.0.1 Additionally, we use the excellent BlenderProc library and our own Blender-toolbox. The pipeline is shown in the figure below, for more details we refer to the paper.
- run
cd data-generation && bash setup.shto install blender and the dependencies, as well as to download all the assets (might take a while).
- to create a scene, run
blender-3.0.1-linux-x64/./blender --python data-generation/towel/towel/generate_towel_scene.pyin the data-generation folder of this repository. Blender will now open and you should see the scene. - to generate the entire dataset used in this work, run
/blender-3.0.1-linux-x64/./blender --python data-generation/towel/towel/generate_dataset.py -- --amount_of_samples 30000 --resolution 256 --start_seed 0. By default the dataset will be created inhome/datasetsbut this can be changed if desired.
The keypoint detection code can be found here: https://github.com/tlpss/keypoint-detection
- git clone the repo at the specific commit using
git clone https://github.com/tlpss/keypoint-detection && git checkout c793f3cf6d803d942054a36ae9b44c410cffa2b3in thekeypointsfolder. - Follow the instructions at https://github.com/tlpss/keypoint-detection to install the dependencies.
- To train on the dataset created:
python /keypoints/keypoint-detection/keypoint_detection/train/train.py --ap_epoch_freq=5 --ap_epoch_start=5 --auto_select_gpus=True --backbone_type=Unet --batch_size=64 --early_stopping_relative_threshold=-0.01 --gpus=1 --heatmap_sigma=2 --image_dataset_path=<path-to-dataset> --json_dataset_path=<path-to-dataset>/dataset.json --keypoint_channel_max_keypoints=4 --keypoint_channels=corner_keypoints_visible --learning_rate=0.0003 --log_every_n_steps=20 --lr_scheduler_relative_threshold=0.005 --max_epochs=16 --maximal_gt_keypoint_pixel_distances="2 4" --minimal_keypoint_extraction_pixel_distance=30 --n_channels=32 --n_downsampling_layers=2 --n_resnet_blocks=16 --num_sanity_val_steps=1 --num_workers=4 --precision=32 --validation_split_ratio=0.1 --wandb_entity <your-wandb-profile> - All training details, as well as the model checkpoints can be found at https://wandb.ai/tlips/icra-2022-workshop/runs/2qand21y?workspace=user-tlips.
- To use the trained model for inference, see below.
- This work uses a UR3e, ZED2i and Robotiq-2F85 gripper. If you have different hardware, you will need to handpick the relevant parts of the code in the
robot/folder. - print 2 fingers using Flexfill-98 TPU filament. The .stl file can be found under
static/cad/ - pip install opencv2:
pip install opencv-contrib-python==4.5.5.64in the conda environment of the keypoint detection code - clone the unreleased dependencies using
vcs import robot < robot/dependencies.repos - pip install them in the same python environment as the keypoint detection code
- determine your camera extrinsics using an Aruco marker and the
marker_pose_to_pickle.pyfile, this will create amarker.picklefile. - Measure the position of your marker to the robot base frame manually and update the code at line 11 of
robot/robot_script.py. Orientations are not supported so make sure to allign the marker to the base frame of the robot. - to manually mark the keypoints (to test the trajectories):
python robot/manual_keypoints.py - to detect the keypoints using the pretrained weights (
model.ckpt) and exectute the fold:python robot/detect_keypoints_and_fold.py
All images from the evaluation can be found here.
@inproceedings{lips2022synthkeypoints,
title={Learning Keypoints from Synthetic Data for Robotic Cloth Folding},
author={Lips, Thomas and De Gusseme, Victor-Louis and others},
journal={2nd workshop on Representing and Manipulating Deformable Objects - ICRA},
year={2022}
}
![pre-commit-ci[bot] avatar](https://avatars.githubusercontent.com/in/68672?v=4 "pre-commit-ci[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent