.h5 file not found after preprocessing (motion correction) step about ciatah HOT 16 OPEN

Oh it did work now, thank you so much! There are a few minor errors that appeared (please see below) but the cell sorting interface seems to be otherwise working fine. Thanks again for taking the time to help with this!

from ciatah.

Hi, hope all is well. To help with this, could you:

• go into the processing_info sub-folder located in the folder where your movie is and send over the output of the .*_preprocess.log file located there?
• run disp(ciapkg.version); in the MATLAB command window and let me know what the output is? Since your errors say calciumImagingAnalysis class whereas most recent version should be ciatah.

As a heads up:

• It is preferred if you perform spatial filtering after the motion correction, as the motion correction methods/algorithm uses a different spatial filter to enhance features in the movie that can be lost by doing another set of filtering beforehand, thus reducing accuracy of alignment.
• CIAtah can downsample Inscopix ISXD files and convert directly to HDF5 in batch (preferred), see https://bahanonu.github.io/ciatah/pipeline_detailed/#converting-inscopix-isxd-files-to-hdf5.

from ciatah.

Hi, thanks a lot for your answer and for your advice regarding spatial filtering, I will try that out next time!

Regarding the version of the package, here is the output :
verStr =

'v3.26.2
 20210325200706'

v3.26.2

In fact, at first I used Ciatah but then I tried the CalciumAnalysisImaging version to see if maybe there was an incompatibility issue between Ciatah and my Matlab version which was older (2017), but the errors are the same in both cases. Here is a link to the log file for a video I tried preprocessing with Ciatah: https://file.re/2021/04/09/20210409094621preprocess/

Thanks again and best wishes

from ciatah.

Hi, thanks a lot for your answer and for your advice regarding spatial filtering, I will try that out next time!

Regarding the version of the package, here is the output :
verStr =

'v3.26.2
20210325200706'
v3.26.2

In fact, at first I used Ciatah but then I tried the CalciumAnalysisImaging version to see if maybe there was an incompatibility issue between Ciatah and my Matlab version which was older (2017), but the errors are the same in both cases. Here is the log file :
20210409094621preprocess.txt

Thanks and best wishes

from ciatah.

@rl72, it looks like both input and output HDF5 dataset names are blank, which is causing the HDF5 save function to fail, e.g. see below in the log file you sent:

                   datasetName: [1×0 char]
             outputDatasetName: [1×0 char]

After creating the CIAtah object (e.g. obj = ciatah;) either run obj.setMovieInfo and ensure the 3rd option is /1 or run the below commands:

obj.outputDatasetName = '/1';
obj.inputDatasetName = '/1';

This should allow you to run modelPreprocessMovieFunction and get movies output fine. Let me know if that works on your end.

from ciatah.

Hi Biafra,

Thanks a lot for your input, everything worked out in the end nicely! I ran CNMFe and EXTRACT smoothly, but I noticed that when running EXTRACT through the CIAtah pipeline, it was possible to modify only a few parameters (the essential ones, such as choosing between GPU/CPU, setting the SNR for cell identification, the number of partitions etc) so I was wondering if there was a way to set more advanced parameters to run EXTRACT in CIAtah?

Thanks and best wishes

from ciatah.

@rl72 Great! That's good to hear.

re: EXTRACT, yep, I was planning to push an update soon, similar to the CNMF-e sub-module, to allow access to more (all) advanced parameters.

from ciatah.

Oh that would be amazing indeed, can't wait to try it out when it is updated then !

On the same topic, I tried running the CIAtah cell sorter on the EXTRACT output (which I ran separately in advance). I used the code provided in the EXTRACT repository but I kept receiving the following error messages in a loop for a very long time until it runs out of memory :

@@@@@@@
Index exceeds matrix dimensions.

Error in compareSignalToMovie (line 101)
nPoints = options.inputMovieDims(3);

Error in signalSorter>viewMontage (line 2463)
croppedPeakImages = compareSignalToMovie(inputMovie, inputImage, thisTrace,'getOnlyPeakImages',1,'waitbarOn',0,'extendedCrosshairs',2,'crossHairVal',maxValMovie*options.crossHairPercent,'outlines',1,'signalPeakArray',signalPeakArray,'cropSize',cropSizeLength,'addPadding',1,'outlineVal',NaN,'xCoords',xCoords,'yCoords',yCoords,'inputDatasetName',options.inputDatasetName,'inputMovieDims',options.inputMovieDims,'hdf5Fid',options.hdf5Fid,'keepFileOpen',options.keepFileOpen);

Error in signalSorter>chooseSignals (line 981)
[objCutImagesCollection{i}, ~] = viewMontage(options.inputMovie,thisImage,options,thisTrace,[signalPeakIdx{i}],minValMovie,maxValMovie,options.cropSizeLength,i,1);

Error in signalSorter (line 608)
[choices, safeExit] = chooseSignals(options,1:nSignals, inputImages,inputSignals,objMap, valid, inputStr,tmpDir,sessionID,signalPeakIdx,signalSnr,inputImagesThres,inputImageSizes,peakOutputStat,ROItraces,imgStats,inputSignalSignal,inputSignalNoise,inputImagesBoundaryIndices,signalPeakIdxOriginal,signalPeaksOriginal,instructionStr);
@@@@@@@
@@@@@@@
Index exceeds matrix dimensions.

Error in signalSorter>chooseSignals (line 1039)
title(['imageCorr = ' num2str(round(peakOutputStat.outputMeanImageCorrs(i)*sigDig)/sigDig) 10 'Press Y to go to last sorted signal.'],'FontSize',options.fontSize,'Interpreter','tex');

Error in signalSorter (line 608)
[choices, safeExit] = chooseSignals(options,1:nSignals, inputImages,inputSignals,objMap, valid, inputStr,tmpDir,sessionID,signalPeakIdx,signalSnr,inputImagesThres,inputImageSizes,peakOutputStat,ROItraces,imgStats,inputSignalSignal,inputSignalNoise,inputImagesBoundaryIndices,signalPeakIdxOriginal,signalPeaksOriginal,instructionStr);
@@@@@@@
0 0
cellmap type: max
@@@@@@@
Index exceeds matrix dimensions.

Error in signalSorter>chooseSignals (line 1271)
' | imageCorr = ' num2str(round(peakOutputStat.outputMeanImageCorrs(i)*sigDig)/sigDig) ',' num2str(peakOutputStat.outputMeanImageCorrs2(i))...

Error in signalSorter (line 608)
[choices, safeExit] = chooseSignals(options,1:nSignals, inputImages,inputSignals,objMap, valid, inputStr,tmpDir,sessionID,signalPeakIdx,signalSnr,inputImagesThres,inputImageSizes,peakOutputStat,ROItraces,imgStats,inputSignalSignal,inputSignalNoise,inputImagesBoundaryIndices,signalPeakIdxOriginal,signalPeaksOriginal,instructionStr);
@@@@@@@
All warnings have the state 'off'.
calculating movie min/max...
finding centroids...
Finding cell neighbors: 2|100
Done assigning neighbor IDs...

ans =

'Frame 1/22725'

Would you know why is this happening?

Thanks a lot!

from ciatah.

@rl72 Did you run signalSorter directly from the command line? If so, can you provide the command(s) you used to run it? That'll help me debug.

from ciatah.

yes, I ran the following :

iopts.inputMovie = M;
iopts.valid = 'neutralStart';
iopts.cropSizeLength = 20;
iopts.cropSize = 20;
iopts.medianFilterTrace = 0;
iopts.subtractMean = 0;
iopts.backgroundGood = [208,229,180]/255;
iopts.backgroundBad = [244,166,166]/255;
iopts.backgroundNeutral = repmat(230,[1 3])/255;

[inputImagesSorted, inputSignalsSorted, choices] = signalSorter(output.spatial_weights, output.temporal_weights', 'options',iopts);

from ciatah.

Thanks. Can you send the following info:

• Matlab version.
• Size of M, e.g. disp(size(M));
• Output on the command window before the errors begin.

from ciatah.

Yes of course :

• Matlab R2018b
• disp(size(M));
  1 11
• and as for the output before error starts, please find the screenshots attached
  
  

from ciatah.

Great, thanks. Looks like M is not in the correct format (x y nFrames). Is that a path to a movie? If so, give the full path rather than just the filename. Else if it is the movie itself, the dimensions do not match the output from EXTRACT.

from ciatah.

If the full path of my file is "D:\lumi_data\data_day4\trial.h5", should I run M as = ('D:\lumi_data\data_day4\trial.h5:/1') ? If yes, it is not working either.

from ciatah.

Try below and let me know if that works for you. If you run into issues, run h5disp('D:\lumi_data\data_day4\trial.h5') to make sure dataset name is /1.

iopts.inputMovie = 'D:\lumi_data\data_day4\trial.h5';
iopts.readMovieChunks = 1; % Binary: 1 = read movie from HDD, 0 = load entire movie into RAM
iopts.inputDatasetName = '/1';

CIAtah does not accept filename and HDF5 dataset name (as with EXTRACT currently) as a single string but has separate variables for each (to avoid certain issues that can arise).

from ciatah.

Great! Glad to hear it is working. Let me know if any other questions come up.

Those warnings should not affect function, but I'll double check in my copy of 2018b (normally testing in Matlab 2019+ these days) to see if they can be removed.

from ciatah.