Code Monkey home page Code Monkey logo

ikflow's Introduction

IKFlow

Normalizing flows for Inverse Kinematics. Open source implementation to the paper "IKFlow: Generating Diverse Inverse Kinematics Solutions"

arxiv.org

Runtime curve for getting exact IK solutions for the Franka Panda (maximum positional/rotational error: 1mm, .572 deg):

alt text

Setup - inference only

git clone https://github.com/jstmn/ikflow.git && cd ikflow
poetry install --without dev
poetry shell

Setup - inference, training, and visualization

The following section outlines the setup procedures required to run the visualizer that this project uses. The only supported OS is Ubuntu. Visualization may work on Mac and Windows, I haven't tried it though. For Ubuntu, there are different system wide dependencies for Ubuntu > 21 and Ubuntu < 21. For example, qt5-default is not in the apt repository for Ubuntu 21.0+ so can't be installed. See https://askubuntu.com/questions/1335184/qt5-default-not-in-ubuntu-21-04.

Ubuntu >= 21.04

sudo apt-get install -y qtbase5-dev qtchooser qt5-qmake qtbase5-dev-tools libosmesa6 build-essential qtcreator
export PYOPENGL_PLATFORM=osmesa # this needs to be run every time you run a visualization script in a new terminal - annoying, I know

Ubuntu <= 20.x.y

(This includes 20.04 LTS, 18.04 LTS, ...)

sudo apt-get install -y qt5-default build-essential qtcreator

Lastly, install with pip:

git clone https://github.com/jstmn/ikflow.git && cd ikflow
poetry install --with dev
poetry shell

Getting started

> Example 1: Use IKFlow to generate approximate IK solutions for the Franka Panda

Evaluate a pretrained IKFlow model for the Franka Panda arm. Note that the value for model_name - in this case panda__full__lp191_5.25m should match an entry in model_descriptions.yaml

python scripts/evaluate.py --testset_size=500 --model_name=panda__full__lp191_5.25m

> Example 2: Use IKFlow to generate exact IK solutions for the Franka Panda

Additional examples are provided in examples/example.py. This file includes examples of collision checking and pose error calculation, among other utilities.

ik_solver, _ = get_ik_solver("panda__full__lp191_5.25m")
target_poses = torch.tensor(
    [
        [0.25, 0, 0.5, 1, 0, 0, 0],
        [0.35, 0, 0.5, 1, 0, 0, 0],
        [0.45, 0, 0.5, 1, 0, 0, 0],
    ],
    device=device,
)
solutions, _ = ik_solver.generate_exact_ik_solutions(target_poses)

> Example 3: Visualize the solutions returned by the fetch_arm__large__mh186_9.25m model

Run the following:

python scripts/visualize.py --model_name=fetch_arm__large__mh186_9.25m --demo_name=oscillate_target

ikflow solutions for oscillating target pose

Run an interactive notebook: jupyter notebook notebooks/robot_visualizations.ipynb

Notes

This project uses the w,x,y,z format for quaternions. That is all.

Training new models

The training code uses Pytorch Lightning to setup and perform the training and Weights and Biases ('wandb') to track training runs and experiments. WandB isn't required for training but it's what this project is designed around. Changing the code to use Tensorboard should be straightforward (so feel free to put in a pull request for this if you want it :)).

First, create a dataset for the robot:

python scripts/build_dataset.py --robot_name=panda --training_set_size=25000000 --only_non_self_colliding

Then start a training run:

# Login to wandb account - Only needs to be run once
wandb login

# Set wandb project name and entity
export WANDB_PROJECT=ikflow 
export WANDB_ENTITY=<your wandb entity name>

python scripts/train.py \
    --robot_name=panda \
    --nb_nodes=12 \
    --batch_size=128 \
    --learning_rate=0.0005

Common errors

  1. GLUT font retrieval function when running a visualizer. Run export PYOPENGL_PLATFORM=osmesa and then try again. See https://bytemeta.vip/repo/MPI-IS/mesh/issues/66
Traceback (most recent call last):
  File "visualize.py", line 4, in <module>
    from ikflow.visualizations import _3dDemo
  File "/home/jstm/Projects/ikflow/utils/visualizations.py", line 10, in <module>
    from klampt import vis
  File "/home/jstm/Projects/ikflow/venv/lib/python3.8/site-packages/klampt/vis/__init__.py", line 3, in <module>
    from .glprogram import *
  File "/home/jstm/Projects/ikflow/venv/lib/python3.8/site-packages/klampt/vis/glprogram.py", line 11, in <module>
    from .glviewport import GLViewport
  File "/home/jstm/Projects/ikflow/venv/lib/python3.8/site-packages/klampt/vis/glviewport.py", line 8, in <module>
    from . import gldraw
  File "/home/jstm/Projects/ikflow/venv/lib/python3.8/site-packages/klampt/vis/gldraw.py", line 10, in <module>
    from OpenGL import GLUT
  File "/home/jstm/Projects/ikflow/venv/lib/python3.8/site-packages/OpenGL/GLUT/__init__.py", line 5, in <module>
    from OpenGL.GLUT.fonts import *
  File "/home/jstm/Projects/ikflow/venv/lib/python3.8/site-packages/OpenGL/GLUT/fonts.py", line 20, in <module>
    p = platform.getGLUTFontPointer( name )
  File "/home/jstm/Projects/ikflow/venv/lib/python3.8/site-packages/OpenGL/platform/baseplatform.py", line 350, in getGLUTFontPointer
    raise NotImplementedError( 
NotImplementedError: Platform does not define a GLUT font retrieval function
  1. If you get this error: tkinter.TclError: no display name and no $DISPLAY environment variable, add the lines below to the top of ik_solvers.py (anywhere before import matplotlib.pyplot as plt should work).
import matplotlib
matplotlib.use("Agg")

Citation

@ARTICLE{9793576,
  author={Ames, Barrett and Morgan, Jeremy and Konidaris, George},
  journal={IEEE Robotics and Automation Letters}, 
  title={IKFlow: Generating Diverse Inverse Kinematics Solutions}, 
  year={2022},
  volume={7},
  number={3},
  pages={7177-7184},
  doi={10.1109/LRA.2022.3181374}
}

ikflow's People

Contributors

jstmn avatar lcford2 avatar llyg777 avatar xyyeh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

ikflow's Issues

I have a question about the negative logarithmic likelihood function during code training.

I have a question about the negative logarithmic likelihood function during code training. I found that the negative logarithmic likelihood function I was training had negative values.

Is this caused by my lack of normalization in processing the dataset? I did not modify the code for ikflow.

And when I looked at the code, I found that the
neg_log_likeli is defined by this code:

output, jac = self.nn_model.forward(x, c=conditional, jac=True)
zz = torch.sum(output**2, dim=1)
neg_log_likeli = 0.5 * zz - jac
loss = torch.mean(neg_log_likeli)

I didn't see the logarithmic operation for zz and jac here when watching Graph INN, nor did I see the softmax layer in the final output of the model. I have been troubled here for a long time, and I really hope to receive your explanation and assistance from the author.

new robot training

Thank you for your excellent work. If I want to train my own robot, what do I need to prepare

  1. Add your own robot's URDF file
  2. Modify the code to include it in robots. py
  3. Modify the model_ Descriptions.yaml file
  4. Generate Dataset
  5. Training Model
  6. Validate the model
    I have tried the above steps, but there are still many things that seem incorrect. Can you give me some guidance?

Collision capsules for new robot training

Hello

I am trying to start a training for Kuka Iiwa14. I have added the URDF file and created a subclass in jrl / robots.py. But i get this when i try to generate dataset or start a training.

screenshot

Do I necessarily need the collision capsules to start a training now?

Many Additional Links When Add Robot

I want to add new robot to robots.py in jrl, and I have many additional links. I find that additional_link_names only accept one link, what should I do?

Inquire about the training and the installation

Hi Jeremy,

Thank you for providing the official implementation code! Some confusing points came up during applying the code and I am wondering if you can help me.

  1. I am new to poetry and am using Ubuntu 20.04. When solving the dependencies, it always tried to install klampt = "0.9.1.post6" and returned error since there is no supporting version. I have no idea why this happened since it seems to me that the version is fixed to "0.9.1.post5" in the jrl project.
  2. How long will the training takes for the panda robot, for the example, "python scripts/train.py --robot_name=panda --nb_nodes=12 --batch_size=128 --learning_rate=0.0005"? In the paper, it says the training is with a NVIDIA GeForce RTX 2080 Ti graphics card. For me it took several hours for one epoch, but the max epoch is set to 5000. How is that possible to finish a training process? Is there anything that I missed?

I will be greatly appreciated if I can hear from you soon.

Best regards,
Yupu

Encountered Nan problem while training a new robot

Hi, Thank you for your excellent work!

I am trying to use ikflow in my project, and my robot is different from a regular robotic arm. It has three additional floating joints, so it has a total of 9 degrees of freedom.

  • I used my own script to generate the same sample and pose datasets, with shapes of [N, 9] and [N, 7], and checked the results in the simulation.
  • I replaced solution_pose_errors with a simple script I implemented myself, but did not implement collision detection (I simply returned False).
  • I used the following hyperparameters to train the script: --dim_latent_space=15 --nb_nodes=9 --batch_size=256 --learning_rate=0.0005, with all other parameters remaining default.

I often encounter the problem of loss being NaN around the 2nd to 5th epochs now. I saw that you also set a warning of loss being NaN in two places. Do you have any relevant suggestions?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.