I'm wondering if you guys think it would be useful to add a keyboard shortcut to the KeywordManager.
I find it easier to debug/test my actions in the unity player rather than deploy all the way to the HoloLens (or emulator).
So to that end I was thinking of adding ability to trigger a Keyword action by pressing a key on the keyboard.

Clarification on blurb from readme:

NOTE: In the interest of simplicity, this test script calls the PlaneFinding APIs directly from the main Unity thread in Update(). In a real application, the PlaneFinding APIs MUST be called from a background thread in order to avoid stalling the rendering thread and causing a drop in frame rate.

Separate background thread - using something like Loom? Further clarification on background thread, please..

The following directories containe some binary objects that are imported from the HoloToolkit project. They are provided for both x64 and x86. This causes Unity to throw an error as the editor can only support a single architecture. In order to fix this I needed to remove the x64 binaries.

Would it be better to include this content as a git submodule rather than a binary build?
If a binary build is needed then what is the proper way to prevent these errors?

Directories containing x64 binaries...

External/HoloToolkit/Sharing/Tools/Profiler/
External\HoloToolkit\Sharing\Tools\SessionManager
Assets/HoloToolkit/Sharing/Plugins

Scenarios:
Gina is a developer. She wants to extend the GazeManager to provide a property for a field directly in front of the users gaze.
Desired code

40         private Vector3 gazeOrigin; 
41         private Vector3 gazeDirection; 
40         public Vector3 AheadOfGaze{ get { return gazeOrigin+gazeDirection; } }

Problem:

1. By changing the GazeManager Gina needs to propagate this change to all customers using her scripts.
2. Changing parts of the HoloToolkit itself locks Gina out from future updates and requires clients to change their HoloToolkit too (removing the 'drop-in-and-run' factor)

Proposed solution:
Make classes partial. Partial classes allow Gina to extend them using her own scripts and export her packages as a drop in component working on top of the base toolkit.
In GinaGazeManager.cs:

public partial class GazeManager{
 public Vector3 AheadOfGaze{ get { return gazeOrigin+gazeDirection; } }
}

The HoloToolKit VertexLit shaders fail to Build in Unity for the UWP Win 10 HoloLens target when light baking is turned on:

Error building Player: Shader error in 'HoloToolkit/Vertex Lit Configurable': invalid subscript 'texcoord1' at Assets/HoloToolkit/Shaders/VertexLitConfigurable.cginc(67) (on d3d11)

Compiling Vertex program with DIRECTIONAL SHADOWS_OFF LIGHTMAP_ON DIRLIGHTMAP_COMBINED DYNAMICLIGHTMAP_OFF
Platform defines: UNITY_NO_SCREENSPACE_SHADOWS UNITY_ENABLE_REFLECTION_BUFFERS UNITY_PBS_USE_BRDF3 UNITY_HARDWARE_TIER1

Scenario:

Adam is a developer who wants an easy way to move his objects around. Place them where they matter to him.
He wants a quick way to tap on an object, gaze at a different location and then tap again to place the object at that location.

Proposed Solution:

Add a new script called TapToPlaceParent or something similar to either the Input or Utilities folder.

I'd like to write two HoloToolkit menu options that automatically apply the settings specified here:

https://developer.microsoft.com/en-us/windows/holographic/unity_development_overview#Configuring_a_Unity_project_for_HoloLens

I'm thinking something like:

• HoloToolkit
  • Auto Configure Project
  • Auto Configure Scene

In my project, setting DrawVisualMeshes to false gives me an exception. In this case, I'm triggering it from a voice command where I stop the observer and stop the drawing of the meshes like so:

_spatialMM.StopObserver();
_spatialMM.DrawVisualMeshes = false;

And I get this exception:

Exception thrown: 'System.NullReferenceException' in WinRTBridge.winmd
NullReferenceException: Exception of type 'System.NullReferenceException' was thrown.
at WinRTBridge.Utils.ThrowNewNullReferenceException(String message)
at UnityEngineProxy.InternalCalls.PInvokeCalls.Renderer_Set_Custom_PropEnabled(IntPtr param_0, Boolean param_1)
at UnityEngineProxy.InternalCalls.Renderer_Set_Custom_PropEnabled(Object self, Boolean paramValue)
at HoloToolkit.Unity.SpatialMappingManager.UpdateRendering(Boolean Enable)
at GameManager.ToggleObserver(Boolean on)
at GameManager.StartPlacement()
at GUIManager.OnPlayCommnand()
at UnityEngine.Events.InvokableCall.Invoke(Object[] args)
at UnityEngine.Events.InvokableCallList.Invoke(Object[] parameters)
at HoloToolkit.Unity.KeywordManager.KeywordRecognizer_OnPhraseRecognized(PhraseRecognizedEventArgs args)
at UnityEngine.Windows.Speech.PhraseRecognizer.InvokePhraseRecognizedEvent(String text, ConfidenceLevel confidence, SemanticMeaning[] semanticMeanings, Int64 phraseStartFileTime, Int64 phraseDurationTicks)
at UnityEngine.Windows.Speech.PhraseRecognizer.$Invoke6(Int64 instance, Int64* args)
at UnityEngine.Internal.$MethodUtility.InvokeMethod(Int64 instance, Int64* args, IntPtr method)
(Filename: Line: 0)

The program '[4280] HoloToolkit-Unity.exe' has exited with code -1 (0xffffffff).

Right now we've got documentation spread throughout the README and the Wiki. This causes a few problems:

• The front page of our GitHub is polluted with in-depth scripting documentation.
• The wiki documentation is hard to collaborate on.
• Documentation is hard to navigate and doesn't connect well.

After investigating using pages ( see comment below ) I'm recommending we simply reorganize the wiki to look something like this:

• Manual
  • Logical descriptions of systems, like the sections of the wiki we have now
  • Descriptions of prefabs and the exposed editor data, should include screenshots from the editor
  • Tutorials
  • Should contain links to relevant Scripting API sections.
• Scripting API
  • API descriptions of script types and their API
  • All the descriptions that currently live in the README
  • Should contain links to relevant Manual sections.

It'd be nice to auto-generate the Scripting API section someday, which should be possible given some VS XML documentation to Markdown Open Source projects out there.

HoloToolkit has a cursor asset that uses a shader called UI-default.shader in Assets\HoloToolkit\Input\Models\Cursor\UI-Default.shader". As a result, Unity confuses using its own UI\Default shader and using HoloToolkit's custom UI\Default shader.

To fix this, the custom shader should be renamed and appropriately placed in HoloToolkit\Utilities\Shaders folder.

Hi all

Another small change that would make it easier to work straight from the editor.
It would be great to call the "OnSelect" message when in the editor we either right mouse click or press a specific key (enter or space) like in the emulator.
Any thoughts?

-s

Assets/HoloToolkit/Build/Editor/BuildDeployWindow.cs(88,13): warning CS0162: Unreachable code detected

After adding the HoloToolkit to a new Unity project and assigning it to collection of holograms, the inside of my office is rendered in spectacular hot pink. By default, my holograms should be occluded by my office walls. I suspect that the renderer is not locating the occlusion material.

My fix:

1. Set the Renderer Mode to Material, and
2. Drag and drop HoloToolkit/SpatialMapping/Materials/Occlusion on to Render Material.

I suspect reopening with an absolute path to the project will solve the problem.

We should instead use a ternary to ensure proper Mono 3.5 support

Need to move the class out of the HoloToolkit namespace unfortunately. Is that a big concern for anyone?

Scenarios:

1. Gina is a developer. She wants to add a script related to Cursors.
   She should be able to add the script file, prefabs, tests, related materials etc. all into the Assets\HoloToolkit\Input folder aka feature folder. This makes the folder more compartmentalized while contributing.
2. David is a designer and he only wants to keep specific feature folders from the HoloToolkit.
   The folder structure should be modular enough so users can delete not needed ones easily.

Proposed folder structure:

Assets\HoloToolkit
-------------------------> Input
----------------------------------> Input \ Scripts
----------------------------------> Input \ Materials
----------------------------------> Input \ Prefabs
----------------------------------> Input \ Tests
-------------------------> Sharing
----------------------------------> Sharing \ Scripts
----------------------------------> Sharing \ Editor
----------------------------------> Sharing \ Prefabs
----------------------------------> Sharing \ Plugins
----------------------------------> Sharing \ Tests

Other folders will match a similar layout.
Dependencies between folders will be documented.

Right now it's a binary asset, not sure how that happened. We should just pull in the conversion to a text asset.

@jbienz - any idea how you got a binary asset here? Curious so we can try to avoid this in the future, though I should have caught it when reviewing the PR.

The preferred method for getting spatial mapping data into Unity is to use the Windows Dev Portal for capturing a 3D model of the room and then loading the meshes from that model. We're currently updating the Spatial Mapping course to show this new flow and will no longer need the following classes in HoloToolkit for sending meshes to Unity:
RemoteMapping.prefab
MeshSaver.cs
RemoteMappingManager.cs
RemoteMeshSource.cs
RemoteMeshTarget.cs
SimpleMeshSerializer.cs

Make SpatialSound folder match the other folders (ex: SpatialMapping) by adding Scripts folder.
Within Scripts, create UAudioManager folder to make it easy to opt in/out of including the component.

Document the menu in the Wiki.

Monitoring performance and frame rate is a pretty important part in developing a HoloLens app.

I quickly found the FPSDisplay script in the toolkit but there wasn't a lot of documentation on how to use it. So I wrote a prefab to make it really easy to integrate in your scene.

You guys think it would be a good contribution to the project? Maybe also use it in one of the demo/test scenes? If so any suggestions where ?

I wrote a blog post with details and a link to the file not sure if it's OK to post it here but you can find it on my profile page.

The problem is that KeywordManager allows sending keyword-triggered events only to static objects that are already in the scene. Sending events to dynamically instantiated prefabs requires additional custom code, so the proposal is for HoloToolkit to provide this functionality out of the box.

The KeywordManager is set to be set up and potentially started on Start, and stopped and destroyed on destroy. However, Start occurs any time the object is enabled, so it can cause the keyword recognizer to throw errors if the game object is disabled and re-enabled.

The KeywordRecognizer should either be set up and started in Awake (which happens once in object lifetime), destroyed and disabled in OnDisable, or have a check to not create a new KeywordRecognizer if one is already set up.

We currently only support local IP (IPOverUSB) due to lack of HTTPS support. We would like to add secure communication to the REST calls used by this window (the Portal class) so that remote IPs can be exposed out.

based on Holograms-220 from the HolographicAcademy

To use HoloToolkit it is necessary to set the -unsafe compiler option. I'm new to C# so I'm not sure what this means, but I did find the following text on the Unity forums:

"Again, using unsafe doesn't give you much advantages, it just opens up for a huge variety of errors, mistakes and unexpected behaviour. Unless you know exactly what you do and you use pointers sparsely you shouldn't go for an unsafe context. " (http://answers.unity3d.com/questions/804103/how-to-enable-unsafe-and-use-pointers.html)

Is it wise to require unsafe? If this needs to remain then this should be documented in the readme (see link above for documentation)

The plane is never assigned to and the surface data is never built.

This is noticeable if you create a "CursorOffHolograms" model that does not have a "back" side to render. What is happening is the normal when no collision is detected is set in the direction of the user's gaze, this turns the model's face away from the user. The solution is to set the normal in the opposite direction of the user's gaze (back toward the user).

This problem came up for me and has now resulted in others reaching out with questions, e.g. microsoft/MixedRealityToolkit#18

This seems like something that could be laid out in the README.

Just a minor observation: I would recommend considering the use of Time.unscaledDeltaTime in scripts instead of Time.deltaTime. especially input related scripts.

One of the simpler ways for Unity games to pause stuff is to set Time.timeScale = 0. Which stops the calling of FixedUpdate and sets Time.deltaTime = 0. Even when we pause game we still want the HoloLens scripts to work properly specifically input scripts.

One could also put an inspector variable in such scripts so the developer can choose deltaTime or unscaledDeltaTime.

So far for inputs, I only see GazeManager using Time.deltaTime; but I also see UAudioManagerBase and Interpolator script using it as well. Not sure about the Audio stuff, but Interpolator will get effected if I need to pause game and want my pause menu UI to tag along.

I noticed this code in the CustomMessages.cs class.

void InitializeMessageHandlers()
    {
        SharingStage sharingStage = SharingStage.Instance;
        if (sharingStage != null)
        {
            serverConnection = sharingStage.Manager.GetServerConnection();
            connectionAdapter = new NetworkConnectionAdapter();
        }

        connectionAdapter.MessageReceivedCallback += OnMessageReceived;
...

If sharingService is null, we don't instantiate the connectionAdapter and the code will crash when we try and set the MessageRecievedCallback.

Unless I'm missing something, I think we should put the last line of code inside the if statement.

Right now the ManualCameraControl script provides a fly mode control which can make it hard to simulate how you would move if you were in a HL experience.
This looks like an interesting and useful change that should make it easier to design HL app straight from Unity.
I propose to add a new control option so it only allow you move on a X/Z plane and on the Y axis (to simulate squat)
An enum would be added so it’s possible to use the current fly mode control (will be the default option) or use the new walk mode control.
Any thoughts?

The script is not removing any verticies.

Unity will spam the console error log with the text "SendMessage OnGazeEnter has no receiver!" if no method for OnGazeEnter or OnGazeLeave has been created for the object. This can be mitigated by using the SendMessageOptions.DontRequireReceiver option as the second parameter to the SendMessage method on the GameObject.

The utilities in the Build Window seem to have a hardcoded 127.0.0.1 IP address that they use. Need a way to specify the actual IP of my device.

Assets/HoloToolkit/SpatialMapping/Scripts/SpatialMappingComponent/Support/SMBaseAbstract.cs(332,88): error CS1501: No overload for method Update' takes1' arguments

Assets/HoloToolkit/SpatialMapping/Scripts/SpatialMappingComponent/Support/SMBaseAbstract.cs(481,33): error CS1501: No overload for method RequestMeshAsync' takes2' arguments

When Start is called on a SurfacePlane, the plane member is assigned to a new BoundedPlane. Since this method is called after Instantiation by SurfaceMeshesToPlanes you'll find that GameObject is in the right spot, but access the BoundPlane gives you zeroed out information.

There does not seem to be any support for game controllers as of yet. Bluetooth controllers will pair with the Hololens but do not function inside of Unity apps. The same controller will work paired to the computer inside the editor or standalone version of the game, which leads me here for solutions.

I've already run into issues where I've wanted the output build folder from BuildsCommands.BuildForHololens to differ from the hardcoded value. I'm thinking I'll add a new section to the preferences for the HoloToolkit and then we'll have a nice place to configure this and any other data that should be configurable.

I suggest adding the following property to GestureManager.cs:

public GameObject FocusedObject { get { return focusedObject; } }

This allows you to get the currently focused object. In fact the Origami academy tutorial even added this.

Windows 10 supports text to speech via SpeechSynthesizer but this isn't easily used from Unity because of the complexities in converting from SpeechSynthesisStream to AudioClip. I propose a new component called TextToSpeechManager (which I've already written) to do this work for us.

Currently defaults to Control, but recommend we default to Shift.

The issue occurs when trying to mouse look and move with WASD. Particularly the S causes this issue.

Hi guys

Just wondering if someone has been working on making the HoloToolKit MicStreamSelector available from Unity ?
Something we would need here soonish so I thought I should ask in case there's already something available privately.

Thanks in advance
-s

To allow highlighting objects you look at, let the gaze manager send OnGazeEnter/OnGazeLeave message on the objects the user looks at.

Isn't it kind of weird how the tagalongs don't orient towards the camera? If you turn about 90 degrees off, the tagalong will be facing so far away from you it's impossible to access.

I tried putting a simple lookat in the Tagalong's update (and restricted rotation to just Y) but I think it's messing up the visibility check with the bounding box so it floats out of view more often...but I'm not totally sure.

Any best practices for keeping a tagalong facing the player--or is this something we need to fix in the tagalong scripts?

Scenario:

By adding the SetFocusPointForFrame to the GazeManager, we will help stabilize the hologram the user is currently gazing at.

We should have components to handle rendering and creating colliders for spatial mapping.

These components should be usable by default but customizable to what most people need. Additionally, we should have heuristics to handle Tracking Lost and moving far away from a placed observer intelligently instead of just removing the meshes.

I'd like to add a sub menu to the HoloToolkit Unity menu that allows a single operation for building and deploying to device or the emulator.

I'm thinking something like:

• HoloToolkit
  • Deploy to Device

Here's the process I see:

1. WSA Build from Unity
2. Nuget restore of VS Solution
3. MSBuild build of VS Solution
4. WinAppDeployCmd of appx.

I'd prefer a solution that only required visual studio, but I'm running into these issues:

1. VS cmd line /build doesn't do the nuget restore (the documentation says it does exactly what Build Solution does, but that's a lie).
2. VS cmd line /build doesn't generate a deployable appx, just binaries.
3. VS cmd line /deploy doesn't let me set the deployment target. (There is no documentation I can find for controlling the target via the command line).

Document the sub menu in the wiki.