Support GoPro's superview lens correction about gyroflow HOT 7 CLOSED

There's an open source project that does this: https://github.com/Niek/superview

from gyroflow.

Superview is implemented as of ae8ecdd
It's not 100% correct, because I still see some minor warping, but I think it's pretty usable.

SuperView is essentially full sensor 4:3 video, non-linearly stretched to 16:9
I took 9 angles of a chessboard with a GoPro in both SuperView and Wide mode (using app so the camera stays in exactly the same spot), then I detected chessboard corners on all 18 images and I tried to figure out an algorithm to make points from SuperView match the ones from Wide shot.
I set up a canvas in html, because we can have live preview in VS Code so the prototyping becomes really quick, I tried to figure out an algorithm and then using Wolfram Alpha I converted the formula to the inverse, so we have the calculation both ways.

I attach the development files, maybe someone wants to pick this up and improve or find out why there's still some warping.

The points are not matching perfectly there, but when I made them aligned better - it was actually producing worse RMS when calibrating the lens, I'm not sure why that is

superview-dev.zip

from gyroflow.

+1, would be great to have SuperView checkbox in export settings.
Obviously there are ways to achieve this without gyroflow, but that would be inconvenient and lower quality

from gyroflow.

Optically it's not anamorphic, it's probably just squeezed in the rescaler.

I would GUESS that performing the cal results in a significantly different Fx and Fy in the camera matrix, do you happen to have an example matrix?

If it does, based on the review of the distortion model and how it works, I suspect that handling this should not be difficult at all, with the possible exception of the FOV math being impacted and/or while I have a pretty good idea of how the stabilization is handled intra-frame in frame_transform.rs - I don't know if there's some additional interframe rotation handling I ahven't noticed yet. (I've got an idea for how to handle it, but I want to confirm my understanding of what's going on is correct, will elaborate more either tonight or tomorrow.)

Specifically, instead of deriving Fx and Fy for new_k in https://github.com/gyroflow/gyroflow/blob/master/src/core/undistortion/frame_transform.rs#L58 (which appears to be effectively creating a "virtual camera" that is panned/rotated to compensate for the real camera's rotation) from the original camera, allowing them to continue to be different, set Fx = Fk = ???? (derived from either original Fx or Fk plus your FOV scaling) - setting Fx = Fk for the "virtual camera" when the recording camera has Fx != Fk should handle the nonsquare pixels of the input.

Also likely would handle the Runcam 5 Orange's Superview mode more easily than the workflow outlined in https://www.youtube.com/watch?v=4JL5R-W9ZzU

I don't have a Gopro or Runcam 5 so can't do the cal experiments myself, although I am seriously considering the R5 Orange.

I wonder if we should possibly create a rollup distortion/lens model issue and/or project since we have quite a few different issues in the tracker that are probably best solved together.

I suspect the math provided in the references of https://github.com/Niek/superview may not be correct, but happens to be an "ok-ish" approximation for semi-defishing something.

from gyroflow.

The runcam 5 XV mode is a linear stretching of the pixels and can be corrected by changing the Fx/Fy values (see #71 ). GoPro superview is a bit more complex and is more akin to a horizontal quadratic curve, but that still has to be confirmed and modelled by comparing 4:3 and superview frames of the same subject to each other.

from gyroflow.

Hm, i guess i misunderstood the original issue. I would simpy like to be able to import 4:3 video, stabilize it and export as superview stretched. Can that be implemented?

from gyroflow.

There's no big practical reason for doing that. The purpose of superview is to allow for a 16:9 image to contain the full 4:3 image captured by the sensor for action shots etc. Due to the way undistortion is applied, the result is also nonlinearly stretched in the horizontal axis, meaning rendering in 16:9 works well for 4:3 input. Applying additional superview leads to bad distortion.

One use case would be to re-add the lens distortion (#30) and then apply superview-like stretching, but the only reason would be to emulate the superview-like effect as opposed to making cinematic footage where any distortion is typically unwanted.

from gyroflow.