paninski-lab / yass Goto Github PK

The blocks.rst file in the doc/ folder contains descriptions for each step in the pipeline, we need to improve the description for some of the steps

Hi,

I have a lot of data generated using Neuropixels. I'd like to try YASS, but I'd imagine that the pre-trained spike detection neural network may not work as it did in the initial publication, given that it seemed to have been trained on MEA retina data. How does performance with the neural network compare with just using spike thresholding? Is it difficult to train a new spike detection network on the data I've already sorted, and how well does it generalize?

Remove templates_partial_data from config, that value sets nPortion, which should also be removed. also remove whitening_batchwise

Hi,
I want to process data in chunks of 0.1 sec with a sampling rate of 30000. There are some issues with this:

1. If I set batch_size:3000 in the yaml file it is ignored.
2. If I calculate the max_memory such that batch_size=3000 then this works but I get an error NameError: global name 'CONFIG' is not defined line 174 in preprocessing.py: if get_score == 1 and CONFIG.spikes.detection == 'nn':
3. If I side-track the CONFIG error then preprocessing works but then I get following error:
   ValueError: could not broadcast input array from shape (50000,5,10) into shape (1239,5,10) line 69 in score.py: score[counter_batch:(counter_batch+nbuff)] = score_temp

Best,
John

Is there any example where more than one cluster is detected per channel? would be nice to include some of the examples you used for your paper so that people can directly compare the clustering results with other methods.
Thanks!

Hi,

Is there any way currently to manually merge/split clusters or call clusters as good/MUA/noise? Phy's template-gui is a nice interface for this; maybe there could be a mode that allows the output to be viewed with phy?

Thanks,
Will

Templates dimension do not match waveforms dimension, for consistency, they should match.

Support processing time intervals for a single channel, then the second channel and so on:

bp = BatchProcessor('path/to/file.bin', max_memory='1gb', channels=(0,4,5),
                                    mode='single_channel')

for batch, interval, channel in bp:
    print('Data from channels {} in interval {}'.format(channel, interval)

This is going to help us apply per-channel transformations (such as standarization) easily and will help us speed up computations in the future, by running per-channel operations in parallel.

Also, the batch processor is limited to reading batches for all channels in long-formatted data. We can also extend the processor to support selectively reading channels.

bp = BatchProcessor('path/to/file.bin', max_memory='1gb', channels=(0,4,5),
                                    mode='all_batches')

for batch, interval in bp:
    print('Data from channels 0,4,5 in interval {}'.format(interval)

This second batching mode should also support setting max_memory to the size of the entire recording for a single batch.

We are going to reuse the indexer for determining the start/end of each batch.

Peter made some changes in the old pipeline that need to be applied to the new one

For most of the operations, we only need to access a small part of the data (e.g. all observations in channel 1, observations 100-200 in channels 1 and 4). given a file-like/array-like (npy/bin), object we should be able to index easily doing something like:

data = indexer('path_to_big_file.bin', t=(100, 200), channels=(1,4))

The indexer bust be agnostic of the data format file/array and shape long/wide. This means building a function that given t and channels, computes the indexes and fetches the data from disk (we do not want to load the entire thing in memory)

For array-like objects (such as .npy), we can use numpy memmap (https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.memmap.html)

For file-like objects (such as .bin), we can use Python mmap (https://docs.python.org/2/library/mmap.html)

Currently, we use relative imports in many of our modules. Since PEP8 standards prefer absolute imports, we should switch to them wherever possible.

Configure documentation building

Migrate codebase to this repo, remove python notebooks, put examples in yass-examples

Hi,
Just following the example section, when I tried to run yass on the sample data, I got this error.

C:\ProgramData\Anaconda2\envs\yass\lib\site-packages\yass\command_line.py:35: De precationWarning: class Preprocessor is deprecated: (Use function in preprocess module, see examples/preprocess.py) pp = Preprocessor(cfg) INFO:yass.preprocessing:Preprocessing the data in progress... C:\ProgramData\Anaconda2\envs\yass\lib\site-packages\progressbar\bar.py:223: Dep recationWarning: The usage of maxvalis deprecated, please usemax_value ins tead 'max_value instead', DeprecationWarning) 2017-12-09 19:40:12.632881: I C:\tf_jenkins\home\workspace\rel-win\M\windows\PY\ 35\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2 Traceback (most recent call last): File "C:\ProgramData\Anaconda2\envs\yass\Scripts\yass-script.py", line 11, in <module> load_entry_point('yass-algorithm==0.4.dev0', 'console_scripts', 'yass')() File "C:\ProgramData\Anaconda2\envs\yass\lib\site-packages\yass\command_line.p y", line 36, in main score, spike_index_clear, spike_index_collision = pp.process() File "C:\ProgramData\Anaconda2\envs\yass\lib\site-packages\yass\preprocessing. py", line 157, in process BUFF, Time) File "C:\ProgramData\Anaconda2\envs\yass\lib\site-packages\yass\preprocessing. py", line 250, in batch_process self.nnTriage File "C:\ProgramData\Anaconda2\envs\yass\lib\site-packages\yass\neuralnetwork\ detect.py", line 115, in nn_detection idx_clean = triage_prob[spike_index[:,0],spike_index[:,1]] > th_triage IndexError: index 188 is out of bounds for axis 0 with size 4

I'm intrigued by that statment:
Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
Should I build TensorFlow from source? (I installed it with Conda).

In previous versions there was a scaleToSave parameter, we need to remove that and also update the code that was using that parameter

There is a potential problem when we replace the string 'ckpt' with 'yaml' when loading the parameters of the nn. For example, someone stores her nn_detector with the following path: /examples/ckpt/nn_detector.ckpt, YASS will loads /examples/yaml/nn_detector.yaml for parameters and throws an error. Maybe we should look at '.ckpt' instead of 'ckpt' ?

We need to review the phy code generation, there is some error I am getting and also some questions:

• do we need n_step neighbors 2?

since running examples with numpy/mplib code embedded in the docs is trouble, it's better to just include the code snippets in the docs. but since its only code, no results are shown. so i thought about this:

• write all examples in jnb in a separate repo (yass-examples)
• create a script to automatically convert jnb to py and move them to the main repo
• create a script to automatically run all jnb examples so every time you update yass, you don't need manually run all the examples
• embed a link to the jnb in the corresponding example?

If there is no tmp folder in the root folder, it throws an error

Since the output for preprocess.run and process.run was updated, we need to update the docstrings as well

Use noise module functions in augmented spikes

in src/yass/preprocessing.py, around line 315, change it to

idx_remove = np.logical_and(
spike_index_clear[:, 0] > BUFF,
spike_index_clear[:, 0] < (rec.shape[0] - BUFF))

spike_index_clear = spike_index_clear[idx_remove]
score = score[idx_remove]

There are some undocumented parameters

New pipeline has some issues, we need to include the examples in the docs that still use the old pipeline