microsoft / mixed-reality-robot-interaction-demo Goto Github PK

Sample Unity application for Android and Hololens, showing how these devices can be used to interact with a (simulated or real) robot through Mixed Reality. Released as part of the IROS 2020 Tutorial on Mixed Reality and Robotics.

Home Page: https://www.microsoft.com/en-us/research/event/mixed-reality-and-robotics-tutorial-iros-2020/

License: MIT License

C# 97.05% ShaderLab 2.37% HLSL 0.20% GLSL 0.32% Dockerfile 0.03% CMake 0.01% Python 0.02%

mixed-reality-robot-interaction-demo's Introduction

IROS 2020 Mixed Reality and Robotics Tutorial

Demo 1: Robot Interaction through Mixed Reality

Microsoft is hosting a workshop at the IROS 2020 conference on Mixed Reality and Robotics. This repository contains all the code required to run, build and deploy Demo 1. The tutorial was available from October 2020 until January 2021 at https://www.iros2020.org/.

The lecture content of the tutorial is now available as a Microsoft Research Webinar: https://note.microsoft.com/MSR-Webinar-Reality-and-Robotics-Registration-On-Demand.html

The hands-on video walkthroughs for the demos are available on the MSR YouTube channel:

Demo 1: Interaction (the demo that goes with the code in this repo)
Demo 2: Colocalization (the demo that uses the ASA ROS Wrapper and is described in the wiki there)

Demo 1 aims to:

Highlight how the robotic community can benefit from Mixed Reality
Show some features of the HoloLens 2, Azure Spatial Anchors, and other products, services & tools from the Mixed Reality ecosystem.
Serve as a playground and starting point for researchers to use Mixed Reality with their own robots

The demo application allows you to command a simulated or real Clearpath Jackal robot and an industrial robot arm attached to it in MR. There are multiple control modes available. This repo also contains Dockerfiles to build and run the Jackal simulation and navigation stack.

Instructions

Please see the Wiki for detailed instructions. The wiki contains detailed instructions on how to run, build and deploy the demo apps. The videos accompanying this repository are currently available through IROS2020.

Support

If you have questions or issues while working with this repository, please open an issue here.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

mixed-reality-robot-interaction-demo's People

Contributors

Stargazers

Watchers

Forkers

pankhurivanjani sycomix tommywu052 jstiefel arrebagrove tomlogan501 dotnet-yj mrcfschr vksharma17 saratrajput bankbiz ang0-0gna jamiemagee krishx007 ricoguerra cz2011301070

mixed-reality-robot-interaction-demo's Issues

Robot wheels are not responding

Dear Eric,

Thank you for the demo!
I am able to deploy the Hololens app and create ASA anchor successfully. However, I couldn’t send the robot to the waypoint marker.
It seems the wheels are not responding.

From the PowerShell, there’s error saying "No p gain specified for pid. Namespace: /gazebo_ros_control/pid_gains/front_left_wheel".

May I have your guidance on this? Your help is highly appreciated.

Best Regards,
Patt

I can't see anything in hololens2

hello,I'm following the youtube guide,but I can't see anything in my hololens2 except this

I don't why,I uploaded the asa config file but it's still like this

How long is the software available?

I'd love to run your nice tutorial, but at the moment there's no time at all for it. Do I have to worry about this repo and the docker containers being deleted at some point? Or can I safely follow your workshop next year as well? (Once I got a hololens for example).

Thanks in advance!

Hololens Build Errors

Hello, I'm following the Youtube guide to build the app for Hololens. I run into the following error:

InvalidOperationException: Certificate Assets/WSATestCertificate.pfx is expired and cannot be used for a UWP build. To fix this, either delete it or select a different certificate in the player settings.
PostProcessWinRT.CreateTestCertificate () (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessWinRT.cs:1071)
PostProcessWinRT.Process () (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessWinRT.cs:173)
UnityEditor.UWP.BuildPostprocessor.PostProcess (UnityEditor.Modules.BuildPostProcessArgs args) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/ExtensionModule.cs:85)
Rethrow as BuildFailedException: Exception of type 'UnityEditor.Build.BuildFailedException' was thrown.
UnityEditor.UWP.BuildPostprocessor.PostProcess (UnityEditor.Modules.BuildPostProcessArgs args) (at C:/buildslave/unity/build/PlatformDependent/MetroPlayer/Extensions/Managed/ExtensionModule.cs:89)
UnityEditor.Modules.DefaultBuildPostprocessor.PostProcess (UnityEditor.Modules.BuildPostProcessArgs args, UnityEditor.BuildProperties& outProperties) (at <8004fcc221b54f98ba547350ea71d812>:0)
UnityEditor.PostprocessBuildPlayer.Postprocess (UnityEditor.BuildTargetGroup targetGroup, UnityEditor.BuildTarget target, System.String installPath, System.String companyName, System.String productName, System.Int32 width, System.Int32 height, UnityEditor.BuildOptions options, UnityEditor.RuntimeClassRegistry usedClassRegistry, UnityEditor.Build.Reporting.BuildReport report) (at <8004fcc221b54f98ba547350ea71d812>:0)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

How are you using Mixed Reality and robotics together?

Please give us some information about your applications, use-cases, and ideas below! :)

Cannot find ASA anchor

I'm not able to locate any ASA anchors. I followed the instructions, put the base coordinate system to some position on the ground and clicked "Create ASA anchor". The Logger tells me:

CreateSpatialAnchor
Attempting to create anchor
Create a cloud spatial anchor with ID = f8c...

I then exit the application and restart it. When I click on "Find ASA anchor" (and try to look at the same location in the room), I regularly get "CloudManager_ErrorServerError: Service unavailable or unspecified error found" in the Logger. Furthermore, on Azure I don't see any created anchors.

Is there a way to get some more debugging messages? I can see a "Development Console" in the corner of my eye, but I can hardly read anything there.

iOS Unity Packages(json file) required quite some rework

As a curiosity, did you ever think of update testing with the Spot instead of Jackal only?

Replacing the robot with Import URDF

Hello. Thank you for your work.

I wanted to replace the robot in this package with another and tried to follow the tutorial in the Wiki. But I wasn't able to do so with the method described there. Instead, I imported the robot using the URDF Importer package from Unity. I'm wondering if there is any difference between the two methods and if we lose any functionality if we don't use the method described in the Wiki.

Android App Crashes When Opened

Hello. I tried to install the provided android app on my Samsung Galaxy S10, but it crashes soon after opening. Is there any other pre-compiled package available?

I got the apk package from this link: https://github.com/microsoft/mixed-reality-robot-interaction-demo/releases/tag/1.0

Docker Build Errors

Hello. I'm following the steps in Building the docker images for Windows and ran into a couple of errors.

Command RUN sudo apt-get install -y apt-utils fails.
Error message:

Get-Content Dockerfile | docker build - -t jackal_single
failed to get console mode for stdin: The handle is invalid.
[+] Building 6.8s (5/19)
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 2.20kB                                                                             0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for iros2020.azurecr.io/jackal_base:latest                                            4.3s
 => CACHED [ 1/16] FROM iros2020.azurecr.io/jackal_base:latest@sha256:85722d4788841d0b6c096c05ed2f8d482513400d23f  0.0s
 => ERROR [ 2/16] RUN sudo apt-get install -y apt-utils                                                            2.4s
------
 > [ 2/16] RUN sudo apt-get install -y apt-utils:
#5 0.417 mesg: ttyname failed: Inappropriate ioctl for device
#5 0.449 Reading package lists...
#5 1.053 Building dependency tree...
#5 1.160 Reading state information...
#5 1.249 The following additional packages will be installed:
#5 1.249   libapt-inst2.0
#5 1.262 The following NEW packages will be installed:
#5 1.263   apt-utils libapt-inst2.0
#5 1.720 0 upgraded, 2 newly installed, 0 to remove and 291 not upgraded.
#5 1.720 Need to get 261 kB of archives.
#5 1.720 After this operation, 1271 kB of additional disk space will be used.
#5 1.720 Ign:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-inst2.0 amd64 1.6.12ubuntu0.1
#5 1.941 Ign:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12ubuntu0.1
#5 2.195 Err:1 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-inst2.0 amd64 1.6.12ubuntu0.1
#5 2.195   404  Not Found [IP: 91.189.88.142 80]
#5 2.415 Err:2 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12ubuntu0.1
#5 2.415   404  Not Found [IP: 91.189.88.142 80]
#5 2.420 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/a/apt/libapt-inst2.0_1.6.12ubuntu0.1_amd64.deb  404  Not Found [IP: 91.189.88.142 80]
#5 2.420 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/a/apt/apt-utils_1.6.12ubuntu0.1_amd64.deb  404  Not Found [IP: 91.189.88.142 80]
#5 2.420 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
------
executor failed running [/bin/bash --login -c sudo apt-get install -y apt-utils]: exit code: 100

Command RUN apt-get -y install ros-melodic-rosbridge-server fails.
Error message:

Get-Content Dockerfile | docker build - -t jackal_single
failed to get console mode for stdin: The handle is invalid.
[+] Building 10.0s (13/21)
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 2.25kB                                                                             0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for iros2020.azurecr.io/jackal_base:latest                                            4.0s
 => [ 1/18] FROM iros2020.azurecr.io/jackal_base:latest@sha256:85722d4788841d0b6c096c05ed2f8d482513400d23f3a5fe97  0.0s
 => CACHED [ 2/18] RUN sudo apt-get update                                                                         0.0s
 => CACHED [ 3/18] RUN sudo apt-get install -y apt-utils                                                           0.0s
 => CACHED [ 4/18] RUN mkdir -p /home/catkin_ws/src                                                                0.0s
 => CACHED [ 5/18] RUN apt-get update                                                                              0.0s
 => CACHED [ 6/18] RUN apt install -y python3-pip                                                                  0.0s
 => CACHED [ 7/18] RUN apt-get install -y python-pip                                                               0.0s
 => CACHED [ 8/18] RUN pip install autobahn tornado bson pymongo pillow                                            0.0s
 => [ 9/18] RUN apt-get update                                                                                     2.2s
 => ERROR [10/18] RUN apt-get -y install ros-melodic-rosbridge-server                                              3.7s
------
 > [10/18] RUN apt-get -y install ros-melodic-rosbridge-server:
#13 0.542 mesg: ttyname failed: Inappropriate ioctl for device
#13 0.579 Reading package lists...
#13 1.270 Building dependency tree...
#13 1.394 Reading state information...
#13 1.502 The following additional packages will be installed:
#13 1.502   python-backports-abc python-backports.ssl-match-hostname python-bson
#13 1.502   python-bson-ext python-singledispatch python-tornado ros-melodic-rosapi
#13 1.502   ros-melodic-rosauth ros-melodic-rosbridge-library ros-melodic-rosbridge-msgs
#13 1.503 Suggested packages:
#13 1.503   python-mysqldb python-pycurl python-tornado-doc
#13 1.538 The following NEW packages will be installed:
#13 1.538   python-backports-abc python-backports.ssl-match-hostname python-bson
#13 1.538   python-bson-ext python-singledispatch python-tornado ros-melodic-rosapi
#13 1.538   ros-melodic-rosauth ros-melodic-rosbridge-library ros-melodic-rosbridge-msgs
#13 1.538   ros-melodic-rosbridge-server
#13 1.804 0 upgraded, 11 newly installed, 0 to remove and 425 not upgraded.
#13 1.804 Need to get 647 kB of archives.
#13 1.804 After this operation, 5458 kB of additional disk space will be used.
#13 1.804 Err:1 http://packages.ros.org/ros/ubuntu bionic/main amd64 ros-melodic-rosbridge-library amd64 0.11.10-1bionic.20200911.044550
#13 1.804   404  Not Found [IP: 140.211.166.134 80]
#13 1.934 Err:2 http://packages.ros.org/ros/ubuntu bionic/main amd64 ros-melodic-rosapi amd64 0.11.10-1bionic.20200911.045908
#13 1.934   404  Not Found [IP: 140.211.166.134 80]
#13 1.934 Err:3 http://packages.ros.org/ros/ubuntu bionic/main amd64 ros-melodic-rosbridge-msgs amd64 0.11.10-1bionic.20200911.044508
#13 1.934   404  Not Found [IP: 140.211.166.134 80]

Expired WSATest certificate

Hi team!

Encountered an issue when deploying Demo 1 to the Hololens2.
I found I couldn't build through Unity because it resulted in an 'expired WSAT certificate' error. After some searching I discovered that if I simply deleted this certificate in the Assets folder I could continue with the build.

Now when attempting to deploy to the Hololens2 through Visual Studio I recieve an error 'Unable to activate Windows Store app 'MSIROSinteractiondemo..etc 'Access is denied'. It would seem now that trying to delete my way out of the issue has caught up with me!

Any ideas? Happy to recreate the issue and take screenshots if that would help.

Best,
Adam