jeongeun980906 / llm-uncertainty-demo Goto Github PK

This code is official implement of the paper CLARA for the real-world demonstraions, with UR5e and RG2 gripper

For details, please check Project Page or Paper Page

Config 1

Config 2

First, please set your own openai API key in

llm-uncertainty/key/key.txt

Install pytorch in the python environment

pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117

But for the ORIN, please check out docs to install pytorch

pip install openai
pip install spacy
python -m spacy download en_core_web_lg
pip install scipy==1.12.0
pip install ultralytics

For detailed installation in ORIN, please check Installation document for the details.

First, you need to put the inference image in the path

./data/det/ori_[index].jpg

for tabletop image, and

./data/det/human_ori_[index].jpg

for user images. The sample for index 1, 2 is provided.

Then, run the code

cd llm-uncertainty
python3 demo.py --move_robot 0 --index [n]

REMEBER: Press !!Enter key!! after you type in goal or answers!

OWL-VIT dependencies, If you are using small devices, please set --on_board 1 in perception engine. This will run YOLOv8 instead of OWL-ViT.

pip install transformers
pip install mujoco-python-viewer
pip install mujoco
pip install pymodbus

Please download the owl-vit checkpoints if you are using those from hugging_face.

llm-uncertainty/owl_vit_model

launch the robot node (UR5e) (For the installation, please follow UR-ROS2)

ros2 launch rilab_ur_launch ex-ur5e.launch.py ur_type:=ur5e robot_ip:=[robot_ip] launch_rviz:=false reverse_ip:=[your_ip]

Launch realsense node. (For the installation, please follow Realsense)

ros2 launch realsense2_camera rs_launch.py align_depth.enable:=true

Then excecute the robot code

cd llm-uncertainty
python3 perception.py --index [n] --on_board [1 or 0]
python3 demo.py --index [n]

REMEBER: Press !!Enter key!! after you type in goal or answers!