rochefort-lab / fissa Goto Github PK

Hi,
I try to process some of my own data based on the example notebook and I get a the following error when i get to the experiment.separate() line:

Doing region growing and data extraction....

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/tmp/ipykernel_25883/3755209102.py in <module>
      1 output_folder = join(sima_path, 'fissa_suite2p_example')
      2 experiment = fissa.Experiment(images, [rois[:ncells]], output_folder)
----> 3 experiment.separate()

~/anaconda3/envs/fissa/lib/python3.7/site-packages/fissa/core.py in separate(self, redo_prep, redo_sep)
    387         """
    388         # Do data preparation
--> 389         self.separation_prep(redo_prep)
    390         if redo_prep:
    391             redo_sep = True

~/anaconda3/envs/fissa/lib/python3.7/site-packages/fissa/core.py in separation_prep(self, redo)
    325 
    326             # get number of cells
--> 327             nCell = len(results[0][1])
    328 
    329             # predefine data structures

IndexError: list index out of range

This is a freshly installed Fissa (0.7.2) through pip in a new conda environment on Ubuntu. Python = 3.7.12. The version for suite2p is 0.10.3, although the processing was done with an earlier version.
The input is fairly standard, 1 plane (512x512) with 148 rois in the iscell group. The parameters seem to check out: num_rois: 148, len(rois): 148, len(cell_ids): 148

Can you provide a few pointers how i can troubleshoot this?

I am trying to implement your method in Matlab, and I have a few questions. But first I would like to thank you for making incredible well organized and easy to read python code!

In the documentation of neuropil.separation the description of the input S is given:

S : array_like
2d array with signals. S[i,j], j = each signal, i = signal content.
j = 0 is considered the primary signal. (i.e. the somatic signal)

I understood it that the signals/timeseries of each roi is along the first dimension, and the different rois along the second dimension, so that the array is of size nSamples x nRois. However, both the output of the signal extraction method and the treatment of S within the separation function seems to me to have to opposite dimension order.

Also, I am wondering about the normalization of A_sep, the separation coefficient matrix. I understood from the paper and the method description that normalizing the scores would ensure that the signals are always positive. However, when the signals are matched, the original score matrix is used, not the normalized one.

Last, when I run nnmf in Matlab with the same number of components as number of rois ( I use the default values, so there are 5 components in total), I always get a score matrix where one element is 1 and the rest are zero for each roi signal. If I set the components to 4 instead, the score matrix looks more like expected, but the resulting demixed signal for the original roi sometimes looks good and sometimes not. The MATLAB implementation of nnmf seems to be a bit different than the scikit one, for example I can not set the parameter alpha (sparseness parameter). I have no experience with the nnmf method and if you have any ideas, or any knowledge of the matlab implementation, I would be very grateful.

I noticed a problem when installing FISSA with "pip install fissa" from an Anaconda environment under Windows. The installation seems to work smoothly, but once I "import fissa" in Python, I get an error message related to the package "geos" when loading the "shapely" package. I got this on two different Windows computers, so I thought it might also affect others.

The problem could be solved by following the suggestions indicated on this Stack overflow question: https://stackoverflow.com/questions/44398265/install-shapely-oserror-winerror-126-the-specified-module-could-not-be-found

I simply used "conda install -c conda-forge shapely", and it worked afterwards.

This would be a useful feature to have for anyone who already uses SIMA as part of their workflow.

Hello,

I have been trying to implement this toolbox on a windows 10 64bit computer. However, the code hangs in the first parallel pool selection of core.py (approx line 277). I have tracked the error down to the below:

File "C:\ProgramData\Anaconda2\lib\multiprocessing\forking.py", line 358, in get_command_line
is not going to be frozen to produce a Windows executable.''')
RuntimeError:
Attempt to start a new process before the current process
has finished its bootstrapping phase.

        This probably means that you are on Windows and you have
        forgotten to use the proper idiom in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

A similar question has been posted here:
https://stackoverflow.com/questions/18204782/runtimeerror-on-windows-trying-python-multiprocessing

Where do I place the " if name == 'main': " to allow the code to run?

Thanks in advance for your help.

Hello again,

Sorry to bother you again.

I have a short question that might have a quick answer.
Do you know if there is a easy way to change the number of iterations and number of trials for each roi until it reaches convergence?

It's rare but it happens on my data that I cant reach convergence.

Thanks a lot in advance,
Georgios

Binder is a new way of serving Notebooks from a GitHub repository on a remote serve without having to download anything to a local machine.

http://mybinder.org/

Would you please include the code for generating simulated data into the fissa toolbox?
Thanks a lot!

One can easily install a developer version with test dependencies by pip install, but there is no explanation how to install the most up-to-date version from GitHub.

For future updates and programmers it would be good to have a standardized way to push updates to PyPi. Will likely add this to CONTRIBUTING.

Hi everyone! I'm using using FISSA with CNMF from MATLAB. At first, I tested the example data (within fissa_master), and all steps are correct. But when I using my own data, exp.seprate(redo_prep = True) reports ValueError: A LinearRing must have at least 3 coordinate tuples. And details are provided in the following figures. Since I'm totally new to Python, I'm not sure how to deal with it.

Thanks for your time!

The ROIs drawn with the oval (and circular) ImageJ tools are saved in a different format to polygon ROIs. This means we can't use the same method to access the data. We need a solution to this, since many people draw their ROIs with the oval tool (since neurons are often near-circular or oval shaped).

Hi everyone!
I am exporting the data in Matlab, and - according to your documentation - I should have:

• result.cell0.trial0(1,:) which should be the final extracted cell signal
• result.cell0.trial0(2,:) which should be the contaminating signal (sometimes zeroed - what does it mean?)

However, I also have others 3 lines of reulsts which are zeroed or non-zeroed. What these lines of data represents?
Which one is the neuropil?
The same is then also applied on the df_result (obtained by running calc_deltaf(3)).

Furthermore, how you "choose the best nRegions"? - At the moment I am running with nRegions = 4 (given by default?).

Thank you a lot for helping me and for this amazing tool !! (:

The test suite is now significantly out-of-date with the actual FISSA functions. This should be updated.

We have been having quite a few questions on how to export ROIs defined in Matlab to Python. This is currently somewhat explained in the cNMF integration tutorial, but a dedicated example going from MATLAB to Python would be very useful also.

Reportedly more and more labs are using the HDF5 data format for data storage. It might be worth adding an option to handle this format, or at least add a clearer tutorial on how to implement such custom formats (as hdf5 files can be very specific to different labs, it might be tricky to make a general loader).

By the way, we noticed that the roi type must be polygon.
While when the roi type is composite type, fissa runs as well. (but resulting roi region is incorrect.)
I just think might be needed to emphasize the expected roi input type, by adding a warning message.
The best would be also support for composite type roi input, if can be easily done.

Originally posted by @zcq0730 in #66 (comment)

Every time FISSA is run you get a per-cell report about the number of iterations and whether NMF converged. This is not useful in most cases and clutters the code-feecback. Instead, I would like to update it so that FISSA just reports 'all converged' if all is well, and a warning message like 'ROIs 1, 2, and 5 did not converge, consider increasing the number of iterations if this happens for a significant number of cells.'

I am encountering the error after running the below script and getting the error following. The script was working until I had a hard drive failure and had to reinstall everything. I imported FISSA and it says that I have the 1.0.0 installed. I tried importing the 0.5.3 version (using pip install fissa==0.5.3) because the python script I am using was written in 2019 but says there is an issue with the package.

#initiate experiment
exp = fissa.Experiment(tiff_folder,rois_FISSA,output_folder)
exp.separate(redo_prep=True)
Extracting traces:   0%|                                                                                                                   | 0/5 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "", line 1, in
  File "/Users/anaconda3/lib/python3.11/site-packages/fissa/core.py", line 1364, in separate
    self.separation_prep(redo_prep)
  File "/Users/anaconda3/lib/python3.11/site-packages/fissa/core.py", line 1259, in separation_prep
    n_roi = len(outputs[0][1])
                ~~~~~~~^^^
IndexError: list index out of range

Is this likely an error with the version of FISSA or something else?

Expected behavior
Whenever an experiment is defined, it should check if a previous analysis was run and if there is data to be loaded. It should then show the message 'Reloading previously prepared data...'.

Actual behaviour
Even if previous analysis data exists, defining a new experiment now always redoes all of the analysis.

Reason
Currently if numpy.load is called, if allow_pickle is not set to True it will throw an error. In our current code this means that whenever an experiment is defined, all of the analysis will be re-done instead of loaded from the previously generated files. The easy fix is to just add 'allow_pickle=True' as an option.

This seems to be a new feature, and it's there for security reasons:

Allow loading pickled object arrays stored in npy files. Reasons for disallowing pickles include security, as loading pickled data can execute arbitrary code. If pickles are disallowed, loading object arrays will fail. Default: False

Changed in version 1.16.3: Made default False in response to CVE-2019-6446.

So we perhaps should think about fixing it in another way, without somehow breaking backwards compatibility with previously stored files!

My ROIs are manual segmented using ImageJ and exported as a RoiSet.zip file. I used this script to run FISSA, and run into the following error. The ROIs seem to cause problems and cannot be converted to masks.

Traceback (most recent call last):
  File "/media/meng/Data/Meng_Psychedelics/02_code/02_prism_soma_dendrite/neuropil_correction.py", line 81, in <module>
    experiment.separate()
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/fissa/core.py", line 1364, in separate
    self.separation_prep(redo_prep)
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/fissa/core.py", line 1246, in separation_prep
    outputs = Parallel(
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/joblib/parallel.py", line 1061, in __call__
    self.retrieve()
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/joblib/parallel.py", line 938, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 595, in __call__
    return self.func(*args, **kwargs)
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/joblib/parallel.py", line 263, in __call__
    return [func(*args, **kwargs)
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/joblib/parallel.py", line 263, in <listcomp>
    return [func(*args, **kwargs)
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/fissa/core.py", line 209, in extract
    base_masks = datahandler.rois2masks(rois, curdata)
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/fissa/extraction.py", line 124, in rois2masks
    return roitools.rois2masks(rois, cls.get_frame_size(data))
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/fissa/roitools.py", line 478, in rois2masks
    return getmasks(rois, shape)
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/fissa/roitools.py", line 400, in getmasks
    mask = poly2mask(rois[i], shpe)
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/fissa/polygons.py", line 70, in poly2mask
    polygons = _reformat_polygons(polygons)
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/fissa/polygons.py", line 137, in _reformat_polygons
    Polygon(polygons[0])
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/shapely/geometry/polygon.py", line 230, in __new__
    shell = LinearRing(shell)
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/shapely/geometry/polygon.py", line 104, in __new__
    geom = shapely.linearrings(coordinates)
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/shapely/decorators.py", line 77, in wrapped
    return func(*args, **kwargs)
  File "/home/meng/anaconda3/envs/2p/lib/python3.10/site-packages/shapely/creation.py", line 173, in linearrings
    return lib.linearrings(coords, out=out, **kwargs)
ValueError: linearrings: Input operand 0 does not have enough dimensions (has 1, gufunc core with signature (i, d)->() requires 2)
    [Extraction 1/3] Converting ROIs to masks

However, when I load the RoiSet.zip and manually convert it into a list of Boolean masks, it works. It is the very same set of ROIs, though.

Is this a bug about specifically dealing with ImageJ .zip files? Can someone fix it?

According to the documentation, fissa.Experiment.save_to_matlab is supposed to include fields 'df_result' and 'df_raw' in the .mat file if df/f0 was calculated.

But it doesn't actually do that, it omits them.

@bolOSIveRcENTR has fixed this in his fork here
bolOSIveRcENTR@5fde4de

However, we can't just pull this commit as-is, because that implementation will fail to complete save_to_matlab if calc_deltaf isn't run beforehand.

FISSA currently also does not have an easy way to only extract the traces for certain subROIs (i.e. without doing the whole neuropil region extraction etc.), might be worth implementing that.

This could be useful if users want to record e.g. a global background signal, or some local reference signal, or just want to extract the signals from some ROIs without the surrounding neuropils.

Also I am seeing an error when I try and call calc_deltaf which states the below:

Traceback (most recent call last):
File "C:\PostDoc Docs\code\python\fissa\examples\basic_usage_v2.py", line 27, in
experiment.calc_deltaf(freq = 1.3)
File "C:\ProgramData\Anaconda2\lib\site-packages\fissa\core.py", line 442, in calc_deltaf
raw_f0 = deltaf.findBaselineF0(raw_conc, freq)
File "C:\ProgramData\Anaconda2\lib\site-packages\fissa\deltaf.py", line 57, in findBaselineF0
b = scipy.signal.firwin(nfilt, cutoff=fw_base / nyq_rate, window='hamming')
File "C:\ProgramData\Anaconda2\lib\site-packages\scipy\signal\fir_filter_design.py", line 394, in firwin
raise ValueError("Invalid cutoff frequency: frequencies must be "
ValueError: Invalid cutoff frequency: frequencies must be greater than 0 and less than fs/2.

I am guessing my sampling freq (1.3Hz) is too low?

Originally posted by @GrimmSnark in #24 (comment)

We should make FISSA be simultaneously both Python 2 and Python 3 compliant.

We are part of the way there already since we already use the future package.

Here is a cheat sheet on which commands to use which are both Python 2 and 3 compatible.
http://python-future.org/compatible_idioms.html

Hello,

First of all, thank you for developing and publishing this fantastic resource for neuropil subtraction.

I have been trying to implement FISSA to extract responses from cells identified through suite2p. I followed the uploaded Jupyter Notebook tutorials and have checked my suite2p outputs, but whenever I run FISSA, the program cannot move past the 'Doing signal separation....' output when I run 'experiment.separate' (seems to be in an infinite loop as I left my program running for 7+ hours on a relatively small data-set before having to force-kill the program). I checked to make sure I was running the most recent FISSA package (0.7.2) as I noticed a previous user had also posted a similar issue in Spyder and the issue appeared to resolve with the the 0.7 version. Launching the example Jupyter binder on this GitHub for using Suite2p + FISSA on example data also runs into the same issue, so I do not believe the issue to be specific to my data.

Any assistance you could provide would be much appreciated!

Thank you,
pocky12

Our current implementation of Suite2P is not up-to-date. Our link to their installation instructions also no longer works. Should fix both of these.

Currently the datahandler functions come from a script. If one wants to use a home-made datahandler, one has to make their own version of the script and import it.

A much cleaner way to do this would be make the datahandler a class, which can be inhereted, after which the relevant aspects of it can be changed. This can then be passed to the existing Experiment class as it is done for imported scripts at the moment.

Hello FISSA developers,

I just tried FISSA on a tiff stack (8000 frames collected at 15 Hz using jRGECO1a) with ROIs defined in ImageJ. In FISSA result, for every ROI, the experiment.result[c][t][1:4, :] are all 0. Is this something normal?

Thanks!

Hi there,

I was running fissa on suite2p motion-corrected reg_tif files following this instruction. When comparing the result with the subtraction method (F = 0.7*Fneu) I noticed that they were nowhere similar to each other. Having a look at the extracted raw signals by fissa (experiment.raw) and by suite2p (F.npy) of the same roi I noticed that also the raw traces were not similar.

I triple-checked the roi identity and plotted experiment.roi_polys[i_roi, trial][0] to compare with suite2p GUI). It definitely is the same roi. Any idea why the raw traces do no agree?

• python: 3.9.12
• suite2p: 0.11.0
• fissa: 1.0.0
• operating system: Windows 11

Just to be sure, the code is attached here:

images = "./reg_tif"

# Get image size
Lx = ops["Lx"]
Ly = ops["Ly"]

# Generate ROI masks in a format usable by FISSA (in this case, a list of masks)
rois = [np.zeros((Ly, Lx), dtype=bool) for n in range(n_cells)]

for i, n in enumerate(cell_ids):
    # i is the position in cell_ids, and n is the actual cell number
    ypix = stat[n]["ypix"][~stat[n]["overlap"]] # get only the non-overlapping pixels
    xpix = stat[n]["xpix"][~stat[n]["overlap"]]
    rois[i][ypix, xpix] = 1
    
fissa_folder = "../../fissa"
fissa_experiment = fissa.Experiment(images, [rois], fissa_folder)
fissa_experiment.separate()

n_cells and cell_ids are previously obtained from F.npy and iscell.npy.

Firstly, thank you for updating FISSA to fix the previous issue that I posted. However, when I updated to the newest version of FISSA, it no longer seemed to work, due to what seems like a threading issue. Since I like to work on multiple things at once, I would ideally like to dedicate half of my CPU to FISSA while I continue working on other things. I probably did something wrong somewhere without realizing it, but I couldn't find the issue (for now) so I thought I'd post an issue. Thank you in advance.

What I did

I ran the following code on Python (3.7.7, 64-bit) with Numpy (1.18.1) on Windows 10 (1909) using FISSA 0.7.0:

import fissa

rois_location = # ROI zip on local hard drive for our experiment
images_location = # Images on local hard drive for our experiment
output_folder = # Folder on local hard drive

experiment = fissa.Experiment(images_location,rois_location,output_folder)
experiment.separate()

I've also run the above code with the number of cores specified as either 8 (half of my cores) or 1 (single-thread).

What I expected

I expected FISSA to do its thing like it did before the update.

What happened

Python spit out the following error:

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Anaconda3\lib\multiprocessing\pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "C:\Anaconda3\lib\multiprocessing\pool.py", line 44, in mapstar
    return list(map(*args))
  File "C:\Anaconda3\lib\site-packages\fissa\core.py", line 62, in extract_func
    base_masks = datahandler.rois2masks(rois, curdata)
  File "C:\Anaconda3\lib\site-packages\fissa\datahandler.py", line 83, in rois2masks
    rois = roitools.readrois(rois)
  File "C:\Anaconda3\lib\site-packages\fissa\roitools.py", line 344, in readrois
    mask_frame = np.nonzero(rois[i]['mask'])[0][0]
IndexError: index 0 is out of bounds for axis 0 with size 0
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda3\lib\site-packages\fissa\core.py", line 389, in separate
    self.separation_prep(redo_prep)
  File "C:\Anaconda3\lib\site-packages\fissa\core.py", line 319, in separation_prep
    results = pool.map(extract_func, inputs)
  File "C:\Anaconda3\lib\multiprocessing\pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Anaconda3\lib\multiprocessing\pool.py", line 657, in get
    raise self._value
IndexError: index 0 is out of bounds for axis 0 with size 0

The above error occurred both when I did and didn't specify the number of cores.

Interestingly, when I ran FISSA with only 1 core, it spit out a shorter error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda3\lib\site-packages\fissa\core.py", line 389, in separate
    self.separation_prep(redo_prep)
  File "C:\Anaconda3\lib\site-packages\fissa\core.py", line 324, in separation_prep
    results[trial] = extract_func(inputs[trial])
  File "C:\Anaconda3\lib\site-packages\fissa\core.py", line 62, in extract_func
    base_masks = datahandler.rois2masks(rois, curdata)
  File "C:\Anaconda3\lib\site-packages\fissa\datahandler.py", line 83, in rois2masks
    rois = roitools.readrois(rois)
  File "C:\Anaconda3\lib\site-packages\fissa\roitools.py", line 344, in readrois
    mask_frame = np.nonzero(rois[i]['mask'])[0][0]
IndexError: index 0 is out of bounds for axis 0 with size 0

Update

I went back and tested fissa with the example files available on GitHub and it worked fine. It must be something on my end, I'll investigate tomorrow in detail after a good night's rest.

Hello,

I am trying to run FISSA on my Suite2p extracted data and I get pretty fast this error:

ValueError: Wrong ROIs input format: unfamiliar shape.

Do you have any idea of what might be causing this?

Thank you very much in advance for your time and help.

Best,
Georgios

When I was playing around with FISSA a few weeks ago, it wouldn't do anything after I set up the experiment, until I typed in experiment.separate() into the console. However, when I tested it out today, it would start region growing and data extraction, almost always failing to finish even after running it for a few minutes.

What I did:

I ran the following code on a Python (3.7.6, 64-bit) IDE (Spyder 4.0.1) on Windows 10 (1909, latest updates installed) using FISSA (0.6.1):

import fissa

rois_location = # ROI zip on local hard drive, taken from FISSA example folder
images_location = # Images on local hard drive, taken from FISSA example folder
output_folder = # Folder on local hard drive

experiment = fissa.Experiment(images_location,rois_location,output_folder,ncores_preparation=4,ncores_separation=4)

What I expected

I expected the code to set up the class 'experiment' as it had always done a few weeks ago.

What happened

Despite not having typed 'experiment.separate()', FISSA would start the region growing and data extraction all by itself, while getting stuck at signal separation.

Update

As I was originally filing out this issue, I was playing around with FISSA and discovered that the above was caused by the presence of 'preparation.npy' and 'separated.npy' in the output folder. When I deleted those, FISSA wouldn't start region growing and data extraction without me typing experiment.separate(). I'm not sure if it's supposed to do that, but I thought I should file an issue report.

Something even weirder is that when the code is run on Spyder (with the output folder emptied), 9 times out of 10 it gets stuck on the 'Doing signal separation' step. However, whenever I tested this out on the python console that I booted directly from the Anaconda powershell prompt, it would finish within a few seconds. It still has the bug (?) where it starts region growing and data extraction by itself, though, but in the dozen or so times that I've run the code, it's given me an output without getting stuck like Spyder.

I've restarted my PC, I've uninstalled FISSA and reinstalled it, but it doesn't seem to fix the issue of FISSA not running properly on Spyder. I'm not sure if this was a problem with Spyder or with FISSA, but I wanted to post this issue because of the signal separation running by itself, in case it was a bug. This was my first issue posting I've done on GitHub so hopefully I followed the guidelines correctly. If I could help getting this to work, it would be a tremendous help to the lab I'm working for. Thank you in advance!

Suite2P is increasingly used for cell detection, and it would be useful to have a clear and easy to use example of how to combine their toolbox with FISSA.

To explain about Suite2P:

• How to extract the motion corrected data
• How to extract the ROIs

To explain about FISSA:

• How to load the above into FISSA

Currently all extracted and separated data are stored as .npy files. A more common storage format would be HDF5, and for some user cases it might also be useful to not store anything at all.

I propose an optional argument like 'file_format', which can be 'npy', 'HDF5', or None.

This relates to #22 and might solve #81 .