BrokenEye is an application that will allow you to obtain more detailed data from the Tobii Eye Tracker for the best social experience in VR, namely:

• Gaze direction
• Eye convergence
• Pupil diameter
• Blink / Eye openness (more details below)
• Streaming images from cameras!

It also allows you to:

• Save and load calibration data
• Record and playback data from device (this is a regular zip).

• Pimax Crystal
• HP Reverb G2 Omnicept Edition

If your headset also supports Tobii tracking and you would like to add support for it to BrokenEye, please open an issue:

• indicate the full name of the headset
• check the functionality in VRCFaceTracking together with the PimaxCrystalAdvanced plugin (without BrokenEye).

If the plugin works, then there is a high chance of working with BrokenEye.

1. Download the latest version of the application from releases.
2. Just launch the application.

After launching the application you can choose:

1. Which device will it connect to?
2. Open data you have already recorded (or record if you are already connected to the device)
3. Do not forget to enable or disable preview of data and images from cameras, whenever you need it, since previewing will create unnecessary load on your computer.
4. Save or load Eye Tracker calibration data (be sure to calibrate the device before saving!).
5. Server settings for streaming data and images from the camera.
6. OSC server settings to receive eye openness data from another application.
7. OSC server parameters.

All settings are saved automatically.

If you find a bug, please report it in the issues. Please attach a logs and configuration file to the bug report, which can be found in the C:\Users\<USER>\AppData\Roaming\fyne\com.ghostiam.BrokenEye folder.

In order to use in VRChat, you should install VRCFaceTracking and a plugin for it PimaxCrystalAdvanced

First run BrokenEye and then VRCFaceTracking.

Since we can only receive data on blinking from the tracker, to obtain data on eye openness, it is necessary use another application that will send data to BrokenEye via the OSC protocol, for example EyeTrackVR

How to configure EyeTrackVR

There are 2 APIs for receiving data:

• HTTP (only for camera images)
• RAW - data is transmitted over a TCP connection, in JSON format for processed data, and images in raw format.

Images from cameras are transmitted as MJPEG stream, which are available via address:

For the left eye:

http://127.0.0.1:5555/eye/left

For the right eye:

http://127.0.0.1:5555/eye/right

You can also preview images in your browser by going to:

http://127.0.0.1:5555/

The default port is 5555, but it can be changed in the application.

To receive data in RAW format, you need to connect via TCP to the application on port 5555.

The request for data looks like this:

we send:

we get in the loop:

An example in C# for obtaining eye tracking data can be viewed in this file.

{
  "left": {
    "gaze_direction_is_valid": false,
    "gaze_direction": [
      // X
      0,
      // Y
      0,
      // Z
      0
    ],
    "pupil_diameter_is_valid": false,
    "pupil_diameter_mm": -1,
    "pupil_position_on_image_is_valid": false,
    "pupil_position_on_image": [
      // X
      -1,
      // Y
      -1
    ],
    "openness_is_valid": true,
    "openness": 1
  },
  "right": {
    "gaze_direction_is_valid": false,
    "gaze_direction": [
      // X
      0,
      // Y
      0,
      // Z
      0
    ],
    "pupil_diameter_is_valid": false,
    "pupil_diameter_mm": -1,
    "pupil_position_on_image_is_valid": false,
    "pupil_position_on_image": [
      // X
      -1,
      // Y
      -1
    ],
    "openness_is_valid": true,
    "openness": 1
  }
}

Where N is the size of the image in bytes:

N = Width * Height * (Bit per pixel / 8)