see title

Should be tested with SteamVR/OpenVR in desktop env

Implement eye() which is equivalent to 'translate(eyePos)`

This is done to bring more parity with Processing-Android

Need to expand on and fill in the holes on inline documentation and retarget for users rather than developers.

Getting

Uncaught TypeError: Cannot read property 'set' of undefined
    at p5xrViewer.set (p5xr.js:8123)

works fine in setup() but canvas is visible before launching and sketch is running in background before AR is entered.

When VR is exited by clicking the little cross on the corner and when it is entered again by clicking "Enter VR" (without page refreshing), setup() and preload() do not run. Only draw() loop runs on subsequent VR sessions.
This is happening because p5 global object is being made only once. Exiting out of VR session does not remove p5 global instance.

Also, VR continues from where it left off when re entered. This is because the global variables are not reset on the next VR session.

This is a subissue for #75

Currently the model view matrix being applied causes the objects to rotate towards rather than away from the view. I believe that the rotation part of the modelViewMatrix is incorrect.

suite('app', function() {
    let myp5;

    setup(function() {
        console.log(' setup! ');
        // PROBLEM HERE
        myp5 = new p5(); 
        // window.setup = function() {};
        // window.draw = function() {};
        // window.vrGlobals = {};
    });

    teardown(function() {
        myp5.remove();
        // window.setup = undefined;
        // window.draw = undefined;
        // window.vrGlobals = undefined;
        console.log(' teardown! ');
    });

    suite('createVRCanvas()', function() {
        test('p5vr is initialised', function() {
            // createVRCanvas();
            // assert.instanceOf(p5xr.instance, p5vr);
            // p5xr.instance.remove();
        });
    });
    
    suite('setVRBackgroundColor()', function() {
        test('curClearColor is set', function() {
            // createVRCanvas();
            // setVRBackgroundColor();
        });
    });
});

Here, clearly there are 2 empty tests with a setup() which only initialises p5 globally and a teardown() which removes the global p5 object.

When I run the test, the 1st test passes while the 2nd fails.
error message:
"before each" hook for "curClearColor is set"
Cannot read property 'call' of undefined

I tried digging in a lot. The problem lies in myp5 = new p5(); in setup. Even if I create p5 object by instance mode, i get not such error. It is happening only when it is created globally. Every other thing i do in setup does not give the error. Also, there is a point to note that using myp5 = new p5(); actually creates global p5 object properly and is as expected in the 1st test(#35 shows it). However, the error comes in the second test.

@stalgiag Can you have a look at this? #35 cannot be completed without solving this.

Would love to antialias by default with p5vr (maybe both?) in order to help with visual fidelity on mobile screens. Need to test to see if the harm on performance is too significant.

Currently the code that is in _userDraw is run twice, once per eye. This is very much not good. A workaround will be difficult but this is high priority. A lot of the p5 functions are linked directly to rendering so they have to happen once per eye. The problem is that other aspects of the user's code that change in the draw loop will also change twice.

This might be a central problem with VR and p5. There are a few design options that will introduce unwelcome complexity. The question is whether to add the complexity on the part of the users with something like xrDraw() or to implement a solution that can rerender the framebuffer from a different view matrix on the second eye.

Any ideas on this matter are definitely welcome.

No details. Just wanted a note to track this. Gonna start working on now.

intersectsBox(Number width, [Number height], [Number depth], screenX, screenY)
intersectsBox(Number width, [Number height], [Number depth], rayObject ray)
- returns Boolean

Description
Checks whether a ray intersects a box. The dimensions of the box is passed in the first 3 arguments and the box will be applied the same transformations as present while calling the function. The height and depth arguments are optional and will take the value of width if height and depth are not provided. If only depth is not provided, it defaults to height.
The last argument is optional. If it is not present, ray will be emerged from mouse position and so intersection with box can be tested with mouse. If a user wishes to use a custom ray, he/she can call generateRay() and pass it in the last argument.

Example

function setup() {
  createCanvas(400, 400, WEBGL);
}

let rotSpeed = 1, rotAngle = 0;
let stopRotation = false, checkIntersection = false;

function draw() {
  background(220);
  push();
  angleMode(DEGREES);
  rotateZ(rotAngle);
  translate(100, 0, 0);
  if(checkIntersection && intersectsBox(50)) {
    stopRotation = !stopRotation;
    checkIntersection = false;
  }
  box(50);
  pop();
  
  if(!stopRotation)
  rotAngle += rotSpeed;
}

function mouseClicked() {
  checkIntersection = true; 
}

High-level implementation overview

1. Retrieve ray coordinates in view space. If it is from mouse, inverse projection matrix will be used. For custom rays, the local ray origin can be transformed via uMVMatrix present in rayObject and local ray direction can be transformed via uNMatrix.
2. Apply current transformation to box collider.
3. Run ray-obb (oriented bounding boxes) intersection algorithm.

This is just an issue to remember to replace supportsSessionMode() with supportsSession() as per most recent spec so that the non-polyfill path will work in Chrome Canary.

Need to make a way for people to report operation on various hardware. Would like to have some support grid.

I think inheriting p5xr from p5 will make it easier to work with p5xr. Adding features should become easier.
I was doing quite a bit of refactoring. I started out trying to do something like this :

p5.prototype.createVRCanvas = function() {
  this.xr = new p5vr(this);

export default class p5xr {
  constructor(p5_instance) {
    this.p5_instance = p5_instance;

There is no inheritance here but I wanted to see how it affects the codebase. It didn't make much difference and had similar issues as the current one.
Then after some thinking, I am more convinced that actual inheritance is a better solution :

import p5 from 'p5';
export default class p5xr extends p5 {

}

I used the p5 npm package. So, I came into a problem. p5vr , p5ar extends p5xr which in turn extends p5. Now, instead of p5 objects being initialised p5vr or p5ar must be initialized. I cannot stop the p5 initialisation (init.js in p5 code) as that code from npm is not modifiable. Also, we need to know beforehand whether the sketch will be ar/vr to initialise p5ar/p5vr. Currently this is done in preload.
The only hurdle is the initialisation part. If we could modify init.js, then we can appropriately initiliase p5ar/p5vr.

I haven't investigated this at all. Just noticed effects in toonShader manual tests.

View is much more pixelated than is logical. Seems to be some odd upscaling. Not sure requires further investigation.

inside of p5xr.onXRFrame, I am consoling as shown below :

console.log("window innerWidth, innerHeight : ", window.innerWidth, window.innerHeight);
if(p5.instance.width < window.innerWidth * window.devicePixelRatio) {
  let oldWidth = p5.instance.width;
  p5.instance.resizeCanvas(
    window.innerWidth * window.devicePixelRatio,
    window.innerHeight * window.devicePixelRatio
  );
  console.log('p5 Canvas resized from '+oldWidth+' to '+p5.instance.width);
}

let viewport = session.baseLayer.getViewport(view);
console.log("viewport width, height : ", viewport.width, viewport.height);
self.gl.viewport(viewport.x, viewport.y,
  viewport.width, viewport.height);

My original window.innerWidth = 393 and window.innerHeight = 786 but I am getting very huge values of window.innerWidth window.innerHeight, viewport.width viewport.height as shown below :

As a result of this, my viewport is so big that the left eye portion actually covers my entire device screen (right eye is rendered outside my device screen) and the rendered images seemed magnified.

Note: On an another device I tested, this issue does not happen and both the left and right eye are rendered correctly. However, it renders correctly only if the following correction in p5xr.onXRFrame is made :

// if(p5.instance.width < window.innerWidth * window.devicePixelRatio) {
if(p5.instance.width < window.innerWidth) {
  let oldWidth = p5.instance.width;
  p5.instance.resizeCanvas(
    // window.innerWidth * window.devicePixelRatio,
    window.innerWidth,
    // window.innerHeight * window.devicePixelRatio
    window.innerHeight
  );
  console.log('p5 Canvas resized from '+oldWidth+' to '+p5.instance.width);
}

I have done basic setup of mocha for browser setting in this PR #29 .

When I started writing a unit test, I realised a problem. Since p5xr requires global p5 object, I am not sure how can i create one in the test files. Obviously, I can have a setup() function globally but that would not allow me to recreate global p5 objects in individual tests (lets say i want a different setup or draw loop). Also, how can i delete global p5 instances.

intersectsSphere(Number radius, [screenX], [screenY])
intersectsSphere(Number radius, rayObject ray) 
- returns Boolean

Description
Checks whether a ray intersects a sphere. The radius of the sphere is passed in the 1st argument and the sphere will be applied the same transformations as present while calling the function.
The 2nd argument is optional. If it is not present, ray will be emerged from mouse position and so intersection with sphere can be tested with mouse. Ray can be emerged from viewer position in the direction specified by parameters screenX and screenY. ScreenX and screenY should be specified in device normalised form -1 <= screenX, screenY <=1. ScreenX and screenY defaults to 0. If a user wishes to use a custom ray, he/she can call generateRay() and pass it in the 2nd paramenter.

Example

let randomx=[], randomy=[];

function preload() {
  createVRCanvas();
}

function setup() {
  setVRBackgroundColor(200, 0, 150);
  for(let i=0; i<5; ++i) {
    randomx[i] = random(-500, 500);
    randomy[i] = random(-500, 500);
  }
}

function draw() {
  setViewerPosition(0, 0, 400);
  for(let i=0; i<5; ++i) {
    push();
    translate(randomx[i], randomy[i]);
    fill('red');
    if(intersectsSphere(70)) {
      fill('blue');
    }
    sphere(70);
    pop();
  }
}

High-level implementation overview

1. Retrieve ray coordinates in view space. If it is from mouse, inverse projection matrix will be used. For custom rays, the ray can be used directly.
2. Apply current transformation to sphere collider.
3. Run ray-sphere intersection algorithm.

While running the unit tests, p5xr instances are created and removed repeatedly. This was causing the polyfill to reinject always and produced unnecessary warnings.

The most suitable method for adding 2D sketch on a 3D sketch is using p5.Graphics().

I am thinking the similar can be done for VR too. As image() is been called, the offset for eyes should be working as even images undergoes uMVMatrix transformation. However, it does not stick to the screen of the viewer. It has a fixed position in 3D space. To make it fix to viewer screen, I will have to take care of the uMVMatrix transformation.

The following example depicts it :

let pg;
let randomx=[], randomy=[];

function preload() {
  createVRCanvas();
}

function setup() {
  setVRBackgroundColor(200, 0, 150);
  pg = createGraphics(windowWidth, windowHeight);
  for(let i=0; i<5; ++i) {
    randomx[i] = random(-500, 500);
    randomy[i] = random(-500, 500);
  }
}

function draw() {
  setViewerPosition(0, 0, 400);
  for(let i=0; i<5; ++i) {
    push();
    translate(randomx[i], randomy[i]);
    fill('red');
    if(intersectsSphere(70, 0, 0)) {
      fill('blue');
    }
    sphere(70);
    noLoop();
    pop();
  }
  pg.circle(windowWidth/2, windowHeight/2, 50);
  image(pg, -windowWidth/2, -windowHeight/2, windowWidth, windowHeight);
}

There are currently changes to the WebXR API. The current spec is not usable with any available browser. I changed the p5ar class to a stub while waiting for the major changes to go into effect.

There are currently a few places where callbacks are using bind to keep the this context correct with lines like --

this.startSketch.bind(this)

This is undesirable. The p5vr object should just keep a self property. Eventually this should maybe be a global singleton as I don't think VR makes sense with 'instance mode'.

See title. This causes p5 to log a warning message.

I have added a simple unit test for createVRCanvas() in the PR #29 .

suite('createVRCanvas()', function() {
    let myp5;
    test('p5vr is initialised', function() {
        return new Promise(function(resolve, reject) {
            window.setup = function() {
                createVRCanvas();
                console.log(window.p5xr.instance instanceof p5vr);
                resolve();
            };
            window.vrGlobals = {};
            myp5 = new p5();
        }).then(function() {
            assert.instanceOf(p5xr.instance, p5vr);
        });
    });
});

The assertion is failing and also console.log(window.p5xr.instance instanceof p5vr) gives false. Also, I printed window.p5xr.instance and p5vr and their output was as expected.
I'm clueless why it's failing.

related to #68

Use jsartoolkit5 (this is what AR.js uses) to handle marker tracking. This will allow marker based AR on devices that cannot run ARCore.

Already drafting this in a branch, but this issue helps track.

I am manually adding the API Reference to the website. This is okay for now but will eventually need to be extracted from comments in the source. We have JSDocs for generating a documentation website but that is no longer in use.

The todo is to automate the extraction of inline comments, output markdown, add it to the appropriate reference docs.

Right now the library replaces p5.RendererGL.prototype._update so that the model view matrix doesn't get reset. This is unnecessary and bad practice. This can easily be replaced with a p5xr specific function since the redraw loop is controlled by the individual p5xr classes. The replacement function should do everything that _update does except reset the perspective and model view matrix.

createVRCanvas() and createARCanvas() do not currently increment and decrement p5's preload as desired. The goal behavior is that these functions keep p5 from leaving preload until after the user has started the immersive session through a gesture.

In the unit tests, I am automatically clicking the "Enter VR" button like this :

let VRbutton = document.querySelector('header button');
VRbutton.click();

However this does not work. This is happening in the polyfilled version (I did not test for non-polyfill). More specifically, device.requestSession() in onXRButtonClicked() is not working and gives me a domexception error when I catch it. In fact, this also happens when I call the function in the src code itself and not in the test file (while normally serving build p5xr).

The example sketch modified in this PR #52 shows the behaviour. With the aid of a reticle drawn at centre, it is evident that the direction of ray from viewer is not set correctly.

Currently there is a call to translate in the draw loop of most of the manual tests that positions the viewer in a location that I just attempted to make 'feel okay'. This is wrong. The todo here is to figure out what translation feels natural enough so that example sketches from non-immersive p5 sketches look okay from the beginning. This translation should then be accounted for automatically by the library. The xrFrameOfRef itself may need to be modified.

I am trying out the toonShader example. It shows me "VR not found". I tried a lot if things like enabling certain xr flags in chrome. I'm not able to run it? How to make it run?

generateRay(Number x1, Number y1, Number z1, Number x2, Number y2, Number z2) - returns rayObject

rayObject : 
{
  origin: p5.Vector,
  direction: p5.Vector
}

Description
The generateRay() function will create a ray under the transforms present at the time of this function call. The origin of the ray will be at (x1, y1, z1) and it will be directed towards (x2, y2, z2). The length of the line (x1, y1, z1) (x2, y2, z2) does not matter as the 2nd point only determines the direction. It returns a ray object which can be passed to intersectsSphere(), intersectsBox() and intersectsPlane().

This function allows a ray to be created with respect to a transform and can be used everywhere in the code where the transforms could vary.

Example

let vrGlobals = {
  counter: 0
};

let randomx = [], randomy = [];

function preload() {
  createVRCanvas();
}

function setup() {
  setVRBackgroundColor(200, 0, 150);
  for (let i = 0; i < 5; ++i) {
    randomx[i] = random(-500, 500);
    randomy[i] = random(-500, 500);
  }
}

function draw() {
  setViewerPosition(0, 0, 400);
  push();
  translate(randomx[0], randomy[0]);
  fill('green');
  sphere(30);
  rotateZ(millis() / 1000);
  line(0, 0, 0, 1000, 1000, 0);
  let ray = generateRay(0, 0, 0, 1000, 1000, 0);
  pop();
  for (let i = 1; i < 5; ++i) {
    push();
    translate(randomx[i], randomy[i]);
    fill('red');
    if (intersectsSphere(70, ray)) {
      fill('blue');
    }
    sphere(70);
    pop();
  }
}

High-level implementation overview

1. Ray direction is calculated by subtracting (x2, y2, z2) from (x1, y1, z1) and normalised.
2. Ray is transformed under current uMVMatrix.
3. The rayObject is returned.

Plan is to use the same API as Processing-Android.

Already have a branch where I am working this out a bit, but just wanted to document as an issue.

This is an intermediary step to allow people to start playing around until #57 is complete.

Again, due to the 'moving target' nature of the spec and API support, the non-polyfill path for VR support is currently broken on Chrome 74.0.3729.

Note this only happens when served from a remote server while testing on Android 8.0.0 Samsung S7. If I run this with a local server such as http-server, then the polyfill path is taken and the library works as expected.

The bug appears to be related to session.requestAnimationFrame which creates a hard freeze in the non polyfilled API which is what is what being served by that version of Chrome.

Note that in Chrome Dev 75.0.3770 this does not happen. There is another bug related to buffer modification due to the timing of resize that occurs in Chrome Dev 75 that comes from the fix to #19 but this is easily fixable.

Due to the very rapid changes happening to the spec, the recommendation here is to continue to work against the polyfilled spec, which likely means only working with sketches served locally, until the development stabilizes a bit on WebXR side.

Getting this error when using with online editor:
Uncaught (in promise) TypeError: Failed to construct 'XRWebGLLayer': parameter 1 is not of type 'XRSession'

session is incorrect here for some reason when running on editor.p5js.org. As far as I can tell this is the only reason that the manual-test-example/p5vr/basic does not work with the online editor. This can probably be resolved easily enough as the code isn't currently using the xrSession property properly and if it were I don't think this would be happening.

Currently, p5xr only supports p5 global mode.
Also, I am unsure how to use p5 node package instead of manually including p5.js file in assets.

Examples use the WebXRButton class for 'Enter XR' input right now. This is fairly simple and should be replaced with something that users can easily modify later on.

IntersectsPlane([rayObject ray]) - returns p5.Vector (2D)

Description
Checks whether a ray intersects a plane. The plane will be applied the same transformations as present while calling the function. The plane will be the same as calling plane() at the place of this function call but with infinite dimensions.
The last argument is optional. If it is not present, ray will be emerged from mouse position and so intersection with plane is tested with mouse. If a user wishes to use a custom ray, he/she can call generateRay() and pass it in the last argument.

Example

function setup() {
  createCanvas(600, 600, WEBGL);
}

let x=0, y=0;

function draw() {
  background(220);
  push();
  fill('#fae');
  translate(0, 0, -100);
  plane(400, 400);
  if(mouseIsPressed) {
    let offset = intersectsPlane();
    x = offset.x;
    y = offset.y;
  }
  translate(x, y, 0);
  fill('red');
  box(50);
  pop();
}

High-level implementation overview

1. Retrieve ray coordinates in view space. If it is from mouse, inverse projection matrix will be used. For custom rays, the local ray origin can be transformed via uMVMatrix present in rayObject and local ray direction can be transformed via uNMatrix.
2. Apply current transformation to plane.
3. Run ray-plane intersection algorithm and return the intersection point.

As a part of my GSoC project, I would like to implement Interface for setting and releasing AR anchors.
Is anchor implemented in the webxr api or will it be added in future? If it is not, then hit test will have to be used to implement anchor in p5 xr.

ARCore uses anchor on top of trackables and session. Maybe I can do something similar where I can anchor objects on a trackable (giving sticky position on the surface tracked) or an object can be anchored to the session (fixed pose in the world space). Is trackables in any form implemented in p5.xr?

I would like to discuss more about this feature since it is a core part of any AR application.

Also, is this repo the complete source code of p5.xr?

There will be a lot of similarities between p5vr and p5ar classes. They should probably extend a base class.

It would be great to have a function that detects whether a ray cast from the camera origin forward in z-space (p5.instance._renderer.uMVMatrix.mat4[10]) intersects with or comes within a certain distance of a vec3. This may seem slightly out of scope but it is the only way I can think of to offer interactivity. This would stand in for the kind of interaction we can get with mouseX and mouseY in the 2D renderer.

This may take quite a bit of work.

Need to have a few spots where the build is publicly available

Need to make fallback for AR mode for devices that do not support ARCore. Mozilla is currently working on an iOS app that enables WebXR for ARKit devices. I think that the support coverage should be roughly:

Least -> Most robust support:

1. Any device with a camera and accelerometer/gyroscope
   • Able to see objects from p5 sketch and use tilting of device to move camera while displaying camera feed
2. Above requirements plus relatively recent hardward (less than 4 years old).
   • Can run jsartoolkit5.This is what AR.js runs on and will allow us to do marker-based anchors.
   • Can also do above
3. Devices that have ARCore (Android >=8.0) or ARKit (iOS >= 12.0) running WebXR-iOS.
   • Able to do everything above and also setAnchor and detectPlane. Allows for correctly mapped anchors in arbitrary location in space.

see title

I noticed that in unit testing with karma and mocha, p5xr.sessionCheck() is not working. More particularly, in sessionCheck() the promise navigator.xr.requestDevice() never gets resolved and remains pending indefinitely. I tested by opening the karma test server on my mobile and debugged remotely. However, (this is quite strange) when I click debug button on the karma test page on my phone, this issue disappears.

@stalgiag I noticed that parcel-bundler is not a dev-dependency and I think it should be added in package.json as a dev-package. If you agree then I'll add it.