Please explain about positional_encoding.py. about neuraloperator HOT 7 CLOSED

Positional encoding concatenates 2 extra values (an x, y coordinate) to the input channels. So, if you have 10 input channels, you'll have 12 after positional encoding.

Imagine you have a 2D spatial domain measuring 64x64 samples. Run this code real quick (shamelessly stolen from numpy's meshgrid reference) and take a look at the 2x64x64 matrix that is produced:

import numpy as np

nx, ny = (64, 64)
x = np.linspace(0, 1, nx)
y = np.linspace(0, 1, ny)
xv, yv = np.meshgrid(x, y)
grid = np.stack((xv, yv))

The 2 layers of the matrix are the x and y coords. Imagine appending this matrix to the end of your input data. There's your positional encoding!

As for what it does, it helps improve convergence by a noticeable margin from my experiments so far. From what I understand, this is because it gives the FNO a constant reference to grid positions regardless of the physical activity that's happening. This produces more consistent signal spectra in the Fourier layers, hence the improved convergence. But I'm not entirely sure myself so that's just my best guess :)

from neuraloperator.

Positional embedding is critical if you want to do super-resolution and want to provide arbitrary positions of query points

from neuraloperator.

@itskobold Thanks for your reply.
Does this mean that data without Positional Encoding is less accurate than data with Positional Encoding because more explicit positional information is not input at the input layer?
Am I correct in assuming that the grid position here refers to physical quantities within the grid points, such as pressure, flow velocity, vorticity, etc.?

from neuraloperator.

Hi @furankaeru, no problem & apologies for the late reply -

From my experiments, positional encoding has improved network convergence every time by a significant (but not huge) margin. I would say yes, from my understanding, positional encoding improves network convergence because it provides explicit positional information to the FNO.

Your assumption is correct; the grid points concatenated to your input data is extra constant data that "underlies" the physical activity you're modelling regardless of what it is.

from neuraloperator.

@itskobold Thanks for your reply.
I see, that makes sense.
Thank you for taking the time to answer my questions.

from neuraloperator.

Sorry, I had one more question.
Is positional encoding available for Navier-Stokes equations?
From reading the documentation, it doesn't seem to say anything about data loading and positional encoding for the Navier-Stokes equations?
Could you give me a sample code to use positional encoding with the Navier-Stokes equations?

from neuraloperator.

Closing as resolved

from neuraloperator.