cxdcxd / rrs Goto Github PK

Realistic Robotic Simulator for Pouring Liquids

C# 24.26% CMake 0.58% Shell 0.10% C++ 13.43% EmberScript 0.10% Python 4.28% Batchfile 0.01% C 2.16% ShaderLab 2.36% HLSL 0.12% GLSL 0.23% HTML 51.39% Dockerfile 0.01% CSS 0.33% JavaScript 0.67%

ros unity3d robotics liquid nvidia flex mlagents kinova-robotics tum

rrs's Introduction

Realistic Robotic Simulator (RRS) for Pouring Liquids

OS	Kernel Version	ROS Version	Nvidia Driver Version	CUDA Version	Unity Version	Ml Agents
Ubuntu 18.04.06 LTS	5.4.0-58-generic	ROS Melodic	460.27.04	11.2	Unity 2020.3.22f1	Release_17

Ml Agents 17 Package Detail

com.unity.ml-agents (C#) v2.0.0
com.unity.ml-agents.extensions (C#) v0.4.0-preview
ml-agents (Python) v0.26.0
ml-agents-envs (Python) v0.26.0
gym-unity (Python) v0.26.0
Communicator (C#/Python) v1.5.0

System architecure and ROS2Unity bridge

The pdf presentation file: ROS2Unity

Setup and Installation

Install ROS Melodic
Clone this Repository and run the automatic installer

git clone https://github.com/cxdcxd/RRS.git
cd rrs_ros
./install.sh

Download Unity 2020.3.22f1 or above
Open Unity Hub and go to Projects, click ADD and then browse the folder rrs_unity from the downloaded repository and launch it. Make sure the Unity Version is set to 2020.3.22f1 or above
Open the repository in a new terminal and type cd rrs_ros and build the workspace: catkin_make
Source the workspace:

setup ~/rrs_ros/devel/setup.bash

Run example

Step 1: (ROS Side)

ros launch rrs_ros rrs_main.launch

Step 2: (Unity Side)

open the scenes/DemoLiquidPouring
Unity3D -> Play

Step 3: (ROS Side)

ros launch mpc real_movo_sc.launch

Settings (rrs_ros)

open the config file from : (RRS/rrs_ros/src/core/cfg/config.yaml)

ntp_server_host_name: test               //define the ntp server hostname (current RRS is not using ntp ignored)
local_network_address: 127.0.0.1         //local ip address
consul_network_address: 127.0.0.1        //consul ip address
consul_network_mask: 255.255.255.0       //consul network mask
consul_network_port: 8500                //consul network port

Settings (rrs_unity)

open the config file from : (RRS/rrs_unity/Config/config.xml)

<?xml version="1.0" encoding="utf-8"?>
<Config xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <consul_network_address>127.0.0.1</consul_network_address>
  <local_network_address>127.0.0.1</local_network_address>
  <consul_network_port>8500</consul_network_port>
  <consul_network_mask>255.255.255.0</consul_network_mask>
  <ntp_server_host_name>test</ntp_server_host_name> //define the ntp server hostname (current RRS is not using ntp ignored)
  <use_relative_origin>false</use_relative_origin>
</Config>

For RL and Gym Training (No need to ROS Side)

open the Assets\Scenes\PourNetTraining.unity

Virtual Environment

mkdir ~/python-envs
python3 -m venv ~/python-envs/sample-env
source ~/python-envs/sample-env/bin/activate
pip3 install --upgrade pip
pip3 install --upgrade setuptools
deactivate

Tensor Board

tensorboard.exe --logdir=".\results\PourNet-LSTM" --host="0.0.0.0" --port=6006

Train example

 mlagents-learn.exe .\TrainingConfig\TrainingConfig\ppo_curriculum_curiosity_lstm.yaml --env=".\ServerBuild\ServerBuild\LearnPouring" --num-envs=32 --base-port=5000 --run-id="PourNet-LSTM-Left-2"

rrs's People

Contributors

Stargazers

Watchers

rrs's Issues

create the src folder for rrs_ros and ignore the devel and build folders

Add docker image for the ROS parts

Simplify installation with a simple docker image

Add Grippers

Add finger actuators, so then we should be able to control the gripper from the ros

Send Force feedback (Haptic Scenario)

Test the my_grasping with unity this time

Update the readme for a fresh installation guidline

Add the force feedback for the joints (effort feedback)

config file folder problems in Unity (windows or linux) platforms

make sure we are all unity 1.17

Test the multi robot scenario with n robots and n docker images

change the config file location

Compare our bridge with rosSharp using several scenarios

State of the art Methods
1)‌ ROS#
2) ROS2
3) URB

Test Cases
1)‌ Network properties (Packet lost, Delay)
2) Unity Performance Analysis (FPS, RAM, CPU,...)
3) Latency (Serialization/ Deserialization)
4)‌‌ Overal Latency with network
5) Bandwidth (Packet Size)

Scenario
1)‌ One instance (Raw Data [Image,‌Pointcloud, nested objects,...])
2)‌ Pick and Place
3) Multi-Instance (1,2,10,100)

Add the TCGUI for generating the network artifacts

Add Hand and Body Colliders

Add all robot Colliders in Unity

Add Gstreamer for the teleoperation scenario

Add Color, Depth and Segmented Cameras

Color Camera
Depth Camera
Segmented Camera

Remove the NTP system
Check Monitors
Check Namespaces

Update the new navigation package

Rename and update all the namings to LMT
Cleanup the package
Test it with the Unity side

Odometry
GroundThrought
cmd_vel as input

Add the Movo's linear joint actuator and control it from ROS

Add a photorealistic scene with RTX
2)‌ Pick and Place Scenario with Medicine Pack