Code Monkey home page Code Monkey logo

viaproma's Introduction

VIAProMa

A Collaborative Mixed Reality Visualization Framework for Immersive Analytics

VIAProMa Logo

Getting Started

Frontend

Prerequisites

Project Setup

  1. Install Unity and Visual Studio
  2. Clone the project
  3. Start Unity, select open and choose the folder Frontend/VIAProMa in the project files
  4. Once the project has been loaded, go to Window > Asset Store (or press Ctrl + 9). Search and download "PUN 2 - FREE" by Exit Games. To do so, the asset has to be aquired first, then downloaded and then imported.
  5. Once PUN has been downloaded, an import window will appear. Select everything and click on import.
  6. After the successful import of the assets, a window should appear which asks for an app ID.
  7. Create a free Photon Engine accout under https://www.photonengine.com/en/pun.
  8. Register a Photon PUN application in order to generate an app ID.
  9. Enter the app ID in the dialog window in Unity. If the window has not appeared or has already been closed, go to Window > Photon Unity Networking > Highlight Server Settings and enter the app ID in the inspector under "Settings/ App Id Realtime".
  10. Again, go to the Asset Store and download "Photon Voice 2" by Exit Games.
  11. In the import window, select everything that is selectable and click on import. Some parts of the import are not selectable. This is fine since they have already been imported with the Photon PUN 2 package.
  12. In the the dashboard of the Photon Engine account, create and register a new Photon Voice application and copy its app ID.
  13. After the import, go to Window > Photon Unity Networking > Highlight Server Settings and enter the voice app ID under "Settings/ App Id Voice".
  14. Open the scene "Main Scene". A prompt will appear which will ask to import TMP Essentials. Click on the upper button.
  15. (Optional) It makes sense to scale the editor icons down. This can be done under "Gizmos" in the top right of the 3D view. Pull the top most slider next to "3D Icons" down until the icons in the scene have the right size
  16. Run the application once in the editor in order to initialize the collection which keeps track of networked objects.

For Android development, additional steps can be found in ARCore's quickstart guide.

Tests

In addition to the working logic, each of the project's features has been isolated into its own scene which contains a minimal working example for the feature. This way, the functionality can be tested and new developers can experiment with these features in order to learn how they work. The scenes are situated in the folder "Tests".

Backend

The backend folder contains a las2peer project which realizes a RESTful service.

Required Technologies

To install the backend the following technologies are required.

  1. Java 17 (on Windows: make sure that the Java installation is placed in your path variables); if you type java –version in the command line, the output should similar to this:
    java version "1.8.0_231"
    Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
    Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)
    

To develop the backend, you should also install an IDE, e.g. IntelliJ.

Building the Backend

First run ./gradlew clean jar and then ./gradlew clean build to build the backend.

Running the Backend

After the build, execute the corresponding “start_network” script in the “bin” folder of the backend. It contains two scripts “start_network.bat” and “start_network.sh”. On Windows, execute the “start_network.bat” file. On Linux or Mac, first go back to the backend folder and execute ./bin/start_network.sh from there.

Publication

Further information about VIAProMa can be found in the following publication:

Hensen B., Klamma R. (2021) VIAProMa: An Agile Project Management Framework for Mixed Reality. In: De Paolis L.T., Arpaia P., Bourdot P. (eds) Augmented Reality, Virtual Reality, and Computer Graphics. AVR 2021. Lecture Notes in Computer Science, vol 12980. Springer, Cham. https://doi.org/10.1007/978-3-030-87595-4_19

Troubleshooting

Problem: When building the application, errors are shown that the namespace name 'HandJointKind', 'HandMeshObserver' and 'JointPose' are not found.

Solution: Download the Windows 10 SDK 10.0.18362.0. After that, open the Build Settings Window and change the Target SDK Version and Minimum Platform Version to 10.0.18362.0.

Problem: The shared room which was created in one app instance does not appear on the other app instance.

Solution: Make sure that both app instances use the same gameVersion which is specified in the launcher script. (Assets/Scripts/Multiplayer/Launcher.cs) Also ensure that both app instances use the same Photon PUN version. The PUN version can be seen in the Photon server settings. Additionally, both instances must have the same Photon app-IDs in the server settings.

viaproma's People

Contributors

benedikthensen avatar danielbekhter avatar emiliehk avatar julianstaab avatar kritiasthovania avatar paulmweiser avatar sebastianmeinberger avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

uciica rfarrey

viaproma's Issues

Sharing of Connection Lines

Using the Photon library, it should be possible to share the connection lines a user created with the other users in the room, such that they can be used for demonstration and visualization purposes.

Use getters and setters instead of public variables

Go through the implemented classes for the connection lines and think about the "read-write" permissions of each variable.

  • Is the variable only public because it should be visible in the inspector? => make it private and add the [SerializeField] tag in front of it.

  • Is the variable public because another script needs to read it but never changes its value (and is not supposed to change its value)? => use a getter

  • Is the variable public because another script changes its value and something should happen if the value is changed => use a getter and setter; call the code that should happen if the variable is changed from the setter

  • Is the variable public because another script needs to read and write it? => you could use a public variable but I would still prefer a getter and a setter

C# has special getter and setter Keywords which can be used:
https://docs.microsoft.com/de-de/dotnet/csharp/language-reference/keywords/get
https://docs.microsoft.com/de-de/dotnet/csharp/language-reference/keywords/set

Connection Lines Menu

Instead of the current buttons that are floating along the cube, one button called "Connection Lines" should be added to the Main Menu cube that opens a new menu where the current buttons and future features should be added, such that they are not just floating in the air.

DrawMode is Activated by Multiple Buttons

The DrawMode can both be entered by the EnterDrawMode and the EnterSingleDelete button (same for exiting the mode). As pressing these buttons in a random sequence might end up with unwanted results, it should be changed. This could be done by introducing a new mode for deleting.

DrawMode UI

This feature is supposed to enable a drawing mode in the application. The mode can be enabled via a button saying "Enter Line Draw Mode" in the main menu. After clicking on that button it should be replaced with the text "Draw Line". Clicking "Draw Line" results in reverting the button to its original text.

Create a test scene

The test scene should allow for fast room creation for the sake of testing the collaborative mode. Besides, it should provide objects of relevant shapes for testing the pointer object's form and its behavior with movement.

Reuse existing color chooser for connection lines UI

There is an existing color chooser which is e.g. used for the Kanban boards. Maybe we can integrate this into the UI of the connection lines?

The color chooser has a dependency on a visualization. It is probably necessary to remove this dependency, e.g. by introducing a color chosen event to which a second script reacts. This second script can set the color of an assigned object, in this case the visualization or the line.

Delete All Lines

A button should be added, that allows users to delete all lines. To do this use for example tags and delete all instances of the LineRenderer Prefab that are labeled with that tag.

Fix rotation of arrow in room with multiple users

When multiple users are in the same room, only the users own arrow rotation sticks out of game objects it is placed on as it should, but the other arrows not.
The rotation variable of the arrow is not shared.

DrawMode Highlighting

In the Draw Mode, the selected objects should be highlighted, such that the user is aware of which objects he/she selected. This highlighting should be removed once the mode is exited by clicking "Draw Line".

Multiple Issues Highlighted

When selecting more than two issues while in DrawMode every issue that has been selected is highlighted even though the selection has been overwritten by another object. This is confusing, as not only the currently selected objects are highlighted. This can be fixed by reseting the color of an issue when another object becomes the new start or destination object.

Line Visibility Settings

The user should be able to select whether his/her lines are visible for other participants in the room. This can be done via a button in the Connection Lines Menu that activates or deactivates the sharing of lines.

Width Change Menu

In the Connection Lines Menu, it should be possible to choose from two different line width settings. This could for example be implemented with two buttons that show the option of a narrower and wider line and they should be selected when in the Line Draw Mode.

Improve pointer object

Choose a pointer object for which rotation is not going to play a major role, so that it does not have to be rotated to individual users in the game.

Update Connection Lines documentation and move it to the Wiki

Since the connection lines are almost ready to be merged with the master branch of the project, the documentation for them should be checked and updated to account for the new menu structure.

It would also be good to collect the documentation at a central place instead of separate readme files. I created a Wiki for this repository where you can document the results.

Implement placement of windows

Implement a movement functionality for all windows which allows the user to place them somewhere else.

One solution would be to introduce a top bar, e.g. behind the title of the window, which the user can grab to drag the window around. A good idea would be to rotate the window towards the user while dragging. The HoloLens 2 introduces an "Adjust" button on such a top bar which allows the user to rotate and scale the entire window with a bounding box.

Reduce number of GetComponent-calls in Connection Line Menu script

For performance reasons, the usage of GetComponent in the script for the connection line menu should be reduced.
There are two ways to do this:

  1. Reference the component itself in the inspector instead of the GameObject. If you drag-and-drop a GameObject onto a field in the inspector, it will automatically choose the fitting component on the GameObject.
  2. Execute GetComponent in the initialization methods which are only executed once, e.g. in Awake or Start, and cache the result in a new private variable.

Example for 1: instead of
`
[SerializeField] private GameObject button;

private void SomeMethod()
{
button.GetComponent().Enabled = false; // accessing the Interactable component
button.name = "My Button" // accessing the GameObject itself
}
`

you can write:
`
[SerializeField] private Interactable interactable;

private void SomeMethod()
{
interactable.Enabled = false; // accessing the Interactable component
interactable.gameObject.Name = "My Button" // accessing the GameObject itself
}
`

Improved Connection Logic

Via utilizing a bounding box the user should be able to choose a specific edge or corner of an object when creating a new line. This helps avoiding the lines covering the content of issue cards and blocking the view of visualizations.

App bar rotates away from the user on tilted task cards

How to recreate

Pull a task card into the scene and rotate it in a way that the top edge is closer to the user than the bottom edge. This means that the front of the task card points towards the floor. The app bar will rotate by 180 degrees and point away from the user, making it impossible to reach its buttons.

Help for fixing the bug

The app bar has a positioning script. It calculates the nearest face and its normal vector based on the box collider of the bounding box. This is used to place the app bar in front of the bounding box's face and with a rotation that the app bar snaps to the bounding box's face. You can probably add a check to invert the rotation if the calculated normal vector points away from the user.
This can be checked with the dot product between the calculated normal vector and the vector from the camera's position to the object.

Fix position/rotation of share/unshare my gaze button

Share/unshare my gaze button is outside of the main menu cube when it is closed.
When the main menu cube is opened the button aligns correctly on the settings panel of the open cube.
The position/rotation/scale of the share/unshare my gaze button should also be correct when the main menu cube is moved or scaled with the bounding box option.

Color Change Menu

In the Connection Lines Menu, it should be possible to choose from several (for example 9) different colors, that the next line to be created should have. This could for example be one button in the color of the currently selected line color, that when clicked displays all possible colors to choose from. After selecting a color it should collapse back into one button displaying the new line color.

Update Position

When objects are moved in the scene, the connecting line should be moved accordingly as well. To do this the coordinates of the Line Renderer should be set to the new coordinates of the object.

Gain overview over branches

The project has accumulated a lot of branches. We should identify the purpose of each branch and identify those that have already been merged. Please create an overview, e.g. in the Wiki which shows which branches carry valuable additions that should be added to the master.

Move connection line menu to an own window

The connection lines should be accessible in an own window which can be placed somewhere in the scene.

  • Remove the additional side from the main menu cube
  • Create a button on the main menu which opens a separate window
  • Create the separate window and move the controls of the connection line menu to this window

You can take a look at existing windows to find out how they work and how they are created.

Make overlay UI work for more than 2 users

Overlay canvas showing the users nickname, color of arrow and device he is using, works with two users in room, when third user enters room canvas only shows one of the two users information

DrawMode Logic

A script enables a mode in which two objects can be selected. Once the "Enter Line Draw Mode" button is clicked, a method enters this mode. Then the two objects can be selected and once the "Draw Line" button is clicked the following happens: With the coordinates of these objects a line is drawn with the Line Renderer component. Therefore a prefab is instantiated that contains the LineRenderer component and the coordinate of the two objects is entered into it.

Save/Load Lines

Using the same loading/saving system as ViaProMa, drawn lines should be saved and loaded alongside the project.

Fix overlay canvas text

Overlay canvas text should show a list of all users, the device they are using and the name of the object currently being gazed/pointed at, for all users in the room
Currently, the name of the user is being displayed correctly, the correct device and name of the object are missing

Delete Specific Lines

In addition to deleting all lines, specific lines should be selected and deleted. For this, an additional button should be added. When it is clicked it turns to the color red (this is an example, it should be possible to distinguish the states of the button), the user is able to select a specific line and with another click on the same button (caption saying "Delete Line") the corresponding gameobject containing the LineRenderer is destroyed.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.