Understanding Eq. 1 and 2 about omnimotion HOT 8 OPEN

@boxraw-tech Tengyu is correct, if two local points map to the same point in the canonical volume, then they are guaranteed to get the same color and density.

from omnimotion.

Hi Tengyu,

I'm not sure if I fully understand your confusion. I'll try to answer your questions, and let me know if you have further ones.

the occlusion relationship would not change across frames as the camera ray still passes through the same set of points in the same order

the occlusion relationship can change, as two different samples $x_j^m$ and $x_j^n$ may swap orders in frame $j$ (i.e., $x_i^m$ is closer than $x_i^n$ in frame $i$, but with deformation $x_j^m$ can become farther away than $x_j^m$ in frame $j$). And the reason that we always naturally sample points from near to far at $p_i$ in frame $i$ is because we want to compute the flow at the location of $p_i$.

I can try to give an example to explain why OmniMotion can handle occlusions. Let's say $p_i$ is occluded by some other surface in frame $j$ at $p_j$, which means $p_i$ should go to $p_j$ but is occluded. Let's assume that the corresponding surface for $p_i$ is $x_i^{n}$ ($\sigma_i^{n}$ is 1 and all other $\sigma$ on the ray are zero). And let's assume that the corresponding surface for $p_j$ in frame $j$ is $x_j^{l}$, then what happens in this case is that $x_i^{n}$ is mapped to $x_j^{n}$ (which projects into $p_j$), but it is farther away than $x_j^{l}$, and that's how it gets occluded. So occlusion happens when some other points exist in front of the points you are tracking. And the other points do not need to be among the points you sampled at $p_i$.

Why not sampling from a new ray in $j$-th frame and map that to the same canonical space?

This can also work if and only if the two points are cycle consistent (co-visible). But the loss in Eq. 2 can be applied to occluded points as well. We tried the idea of enforcing cycle-consistent points to be mapped to the same canonical space, but it didn't work very well. In fact, what you need is not only enforcing matching points to be closer but also non-matching points to be further away, otherwise a trivial solution would be to make the canonical space infinitely small. But we didn't find a version of this loss to work robustly well either.

Best,
Qianqian

from omnimotion.

Please correct me if my understanding is wrong:

The occlusion relationship can change, as two different samples $x_j^m$ and $x_j^n$ may swap orders in the frame $j$

If $x_i^m$ is closer than $x_i^n$, that means $m\lt n$, and $T_m\cdot\alpha_m=1$ and $T_n\cdot\alpha_n=0$ for both the frames $i$ and $j$. This will not change the occlusion relationship even if the depth order changes between the two frames. Unless you re-order $x_j$ by depth.

it ($x_j^n$) is farther away than $x_j^l$, and that's how it gets occluded

Because both $x_j^l$ and $x_i^l$ map to the same point in the canonical volume, they would get exactly the same color and density right? Consider in a rigid scene, something that was not occluded in frame $i$ ($x_i^n$ is closer than $x_i^l$) is occluded in frame $j$ ($x_j^l$ is closer than $x_j^n$) due to only camera motion. The only way OmniMotion would works is that $x_i^l$ is already in the camera ray, hiding behind $x_i^n$ even though in reality the occluding object should not be in the camera ray in frame $i$. Is my understanding correct? I believe that this is why it is called a quasi-3D representation in the sense that it is geometrically incorrect but suits well for the dense tracking task.

from omnimotion.

@tengyu-liu thanks for asking these questions, I'm also trying to get my head around this.

Because both $x_j^l$ and $x_i^l$ map to the same point in the canonical volume, they would get exactly the same colour and density right?

My understanding is they don't necessarily get the same colour and density as $F_\theta$ is parameterised differently for each frame by $\psi_i$.

from omnimotion.

According to sections 4.1 and 4.2, I believe that $F_\theta$ is independent of the frame. $M_\theta$ is parameterised by $\phi_i$, which gives different $\mathcal{T}_i$ functions for different frames. Since both $x_i^l$ and $x_j^l$ map to the same point in the canonical volume, they are guaranteed to get the same color and density.

from omnimotion.

In section 4.3 it says

density and colour can be written as $(\sigma_k, c_k) = F_\theta (M_\theta(x^k_i ; \psi_i)) $

so it seems perfectly possible to get different colour and density for the same point in different frames.

from omnimotion.

Hi Tengyu,

The first part is correct. However, I think there is some misunderstanding here:

The only way OmniMotion would work is that $x_i^l$ is already in the camera ray, hiding behind $x_i^n$ even though in reality the occluding object should not be in the camera ray in frame i.

I don't understand why that's the only way OmniMotion would work. $x_i^n$ and $x_i^l$ do not need to be on the same ray in frame $i$. They can be at different pixel locations and both of them can be visible. Let me give you an example using the online demo:

In this example, the blue point is occluded by the red point in the second image (let's assume they are at the same pixel location), but their corresponding pixel locations in the first image are different and both of them are visible.

from omnimotion.

Hi @qianqianwang68 @tengyu-liu

I am still trying to wrap my head around the discussion. Could you guys help to clarify them for me, particularly

If $\mathbf{x}_i^{m}$ is closer than $\mathbf{x}_i^{n}$, that means m<n, and $T_m$ $\circ$ $\alpha_m$=1 and $T_n$ $\circ$ $\alpha_n$=0 for both the frames i and j. This will not change the occlusion relationship even if the depth order changes between the two frames. Unless you re-order $\mathbf{x}_j$ by depth.

Following your discussion, assuming m<n, the corresponding surface for $\mathbf{p}_i$ is $\mathbf{x}_i^n$, thus $\sigma_i^n$=1 and the rest is 0 ($\sigma_i^m$=0), isn't $T_m$ $\circ$ $\alpha_m = 0$ and $T_n$ $\circ$ $\alpha_n$=0 since $\alpha_m = 1 - \exp(0)$=0 and since $T_k$ considers the product over the $k-1$ points, thus $T_n$ $\circ$ $\alpha_n$ should be 0 as well.

from omnimotion.