[Bug]: Cannot detrend pairs - ValueError: Chunk shape must not be greater than data shape in any dimension. (512, 512) is not compatible with (3750, 375) about pygmtsar HOT 9 CLOSED

The single function doesn’t save the result on disk.

from pygmtsar.

Single product detrending works fine on helper function.

def plot_sample(idx, wavelength, fit_intercept=True, fit_dem=True, fit_coords=True, resolution_meters=90):
    pair = pairs[idx:idx+1]
    da = sbas.open_grids(pair, 'unwrap')[0]
    # crop correlation grid like to unwrap grid if needed
    corr = sbas.open_grids(pair, 'corr')[0].reindex_like(da)
    label = da.pair.item()
    da_detrend = sbas.detrend(da,
                              wavelength=wavelength,
                              fit_intercept=fit_intercept,
                              fit_dem=fit_dem,
                              fit_coords=fit_coords,
                              resolution_meters=resolution_meters)

    fig = plt.figure(figsize=(16,4), dpi=300)

    ax = fig.add_subplot(1, 4, 1)
    corr.plot.imshow(cmap='gray', vmin=0, vmax=1, ax=ax)
    ax.set_title('Correlation', fontsize=18)

    #zmin, zmax = np.nanquantile(da, [0.001, 0.999])
    ax = fig.add_subplot(1, 4, 2)
    da.plot.imshow(cmap='turbo', ax=ax)
    ax.set_title('Phase, [rad]', fontsize=18)

    #zmin, zmax = np.nanquantile((da - da_detrend), [0.001, 0.999])
    ax = fig.add_subplot(1, 4, 3)
    (da - da_detrend).plot.imshow(cmap='turbo', ax=ax)
    ax.set_title('Trend, [rad]', fontsize=18)

    #zmin, zmax = np.nanquantile(da_detrend, [0.001, 0.999])
    ax = fig.add_subplot(1, 4, 4)
    da_detrend.plot.imshow(cmap='turbo', ax=ax)
    ax.set_title('Detrended, [rad]', fontsize=18)

    plt.suptitle(f'AOI Unwrapped Phase {label.replace(" "," to ")}', fontsize=18)
    plt.tight_layout()
    plt.show()

%%time
plot_sample(1, None, fit_dem=True, fit_intercept=True, fit_coords=True)

from pygmtsar.

ValueError: Chunk shape must not be greater than data shape in any dimension. (512, 512) is not compatible with (3750, 375)

The command datagrid.chunksize = 256 should be called before creating NetCDF files (before unwrapping for this case) to define the chunk size and save the grids with it. Also, you can use a slightly wider unwrapping mask (exactly 512 pixels would be the best size) to resolve the issue. I hope that someday it will be fixed in xarray (like the annoying NetCDF saving notices).

from pygmtsar.

sbas.detrend_parallel(unwraps, resolution_meters=15)

ValueError Traceback (most recent call last)
Cell In[89], line 1
----> 1 sbas.detrend_parallel(unwraps, resolution_meters=15)

File ~/.local/lib/python3.10/site-packages/pygmtsar/SBAS_detrend.py:24, in SBAS_detrend.detrend_parallel(self, pairs, n_jobs, interactive, **kwargs)
21 import joblib
23 # convert pairs (list, array, dataframe) to 2D numpy array
---> 24 pairs = self.pairs(pairs)[['ref', 'rep']].astype(str).values
26 def func(pair, **kwargs):
27 #print (f'**kwargs {kwargs}')
28 grid = self.open_grids([pair], 'unwrap', interactive=False)[0]

File ~/.local/lib/python3.10/site-packages/pygmtsar/SBAS_base.py:426, in SBAS_base.pairs(self, pairs, dates, name)
419 # check that all the pairs produced from the SBAS scenes
420 #dates = list(map(lambda x: x.replace('-',''), self.df.index))
421 #invalid = [pair for pair in pairs.flatten() if pair not in dates]
422 #assert len(invalid) == 0, 'ERROR: found grids for pairs not in the SBAS scenes. Define valid pairs manually.'
424 if not isinstance(pairs, pd.DataFrame):
425 # Convert numpy array to DataFrame
--> 426 pairs = pd.DataFrame(pairs, columns=['ref', 'rep'])
427 # Convert ref and rep columns to datetime format
428 pairs['ref'] = pd.to_datetime(pairs['ref'])

File ~/.local/lib/python3.10/site-packages/pandas/core/frame.py:798, in DataFrame.init(self, data, index, columns, dtype, copy)
790 mgr = arrays_to_mgr(
791 arrays,
792 columns,
(...)
795 typ=manager,
796 )
797 else:
--> 798 mgr = ndarray_to_mgr(
799 data,
800 index,
801 columns,
802 dtype=dtype,
803 copy=copy,
804 typ=manager,
805 )
806 else:
807 mgr = dict_to_mgr(
808 {},
809 index,
(...)
812 typ=manager,
813 )

File ~/.local/lib/python3.10/site-packages/pandas/core/internals/construction.py:315, in ndarray_to_mgr(values, index, columns, dtype, copy, typ)
309 _copy = (
310 copy_on_sanitize
311 if (dtype is None or astype_is_view(values.dtype, dtype))
312 else False
313 )
314 values = np.array(values, copy=_copy)
--> 315 values = _ensure_2d(values)
317 else:
318 # by definition an array here
319 # the dtypes will be coerced to a single dtype
320 values = _prep_ndarraylike(values, copy=copy_on_sanitize)

File ~/.local/lib/python3.10/site-packages/pandas/core/internals/construction.py:570, in _ensure_2d(values)
568 values = values.reshape((values.shape[0], 1))
569 elif values.ndim != 2:
--> 570 raise ValueError(f"Must pass 2-d input. shape={values.shape}")
571 return values

ValueError: Must pass 2-d input. shape=(873, 3750, 375)

from pygmtsar.

So do i have to reprocess unwraps?

from pygmtsar.

In case it takes a long time, I could make a patch to the detrend_parallel() function to define the chunksize within it. Would you like to try and test it?

from pygmtsar.

In case it takes a long time, I could make a patch to the detrend_parallel() function to define the chunksize within it. Would you like to try and test it?

Don't worry, I will redo unwrapping with 512 width. Why does single detrend() function work though?

from pygmtsar.

Minimum slice size based on chunksize fixed the issue.

from pygmtsar.

Fixed in 9133d8a

from pygmtsar.