Maximum Covariance Analysis is slower than one in the nicrie/xmca about xeofs HOT 3 CLOSED

Great catch! You're correct, the issue you mentioned isn't related to dask. The cross-covariance from your data has dimensions 64800 x 10368, which is near the maximum size solvable on a typical laptop using np.linalg.svd. For such large matrices, the randomized SVD from sklearn is recommended as it's faster. That's why it is the default option in xeofs.

However, xmca is still able to handle the datasets with np.linalg.svd, because it first conducts standard PCA on each data set prior to computing the cross-covariance. Given that your data sets have 492 time steps, they can be fully represented by 492 PCs. Thus, when performing MCA in this reduced PCA space, the SVD solution should be swift.

Currently, xeofs doesn't offer this PCA preprocessing. But, good news – it's present in the development branch. If you need it urgently, install the development version:

pip install git+https://github.com/nicrie/xeofs.git@develop

The development version then allows you to specify the number of PCA modes. Since in your case the rank of the input matrices are 492, you can easily provide all PC modes so you don't loose any information by using the PCA solutions as input for MCA. In the development version you would specify it like this:

model = MCA(n_modes=5, standardize=False, use_coslat=True, n_pca_modes=492),
model.fit(data_input1, data_input2, dim='time')

Otherwise, if you don't mind waiting a couple of days more, I'll be releasing a new version with this feature soon.

from xeofs.

Yeah, just what I wanted, thank you. I found this trick, looking forward to the new release

from xeofs.

this feature is now available in the new release v.1.1.0 (see #80 )

from xeofs.