Code Monkey home page Code Monkey logo

m-beau / neuropyxels Goto Github PK

View Code? Open in Web Editor NEW
103.0 3.0 22.0 42.75 MB

NeuroPyxels (npyx) is a python library built for electrophysiologists using Neuropixels electrodes. This package stems from the need of a pythonist who really did not want to transition to MATLAB to work with Neuropixels: it features a suite of core utility functions for loading, processing and plotting Neuropixels data.

License: GNU General Public License v3.0

Python 8.67% Jupyter Notebook 91.33%
python electrophysiology

neuropyxels's Introduction

PyPI Version DOI License Downloads

NeuroPyxels: loading, processing and plotting Neuropixels data in Python

Neuropixels

NeuroPyxels (npyx) is a python library built for electrophysiologists using Neuropixels electrodes. This package results from the needs of a pythonist who really did not want to transition to MATLAB to work with Neuropixels: it features a suite of core utility functions for loading, processing and plotting Neuropixels data.

Any questions or issues?: Create a github issue to get support, or create a pull request. Alternatively, you can email us: maximebeaujeanroch047[at]gmail[dot]com. You can also use the Neuropixels slack workgroup.

⬇️ Installation:

We recommend using a conda environment. Pre-existing packages on a python installation might be incompatible with npyx and break your installation. You can find instructions on setting up a conda environment here.

  conda create -n my_env python=3.10
  conda activate my_env
  pip install npyx
  # optionally (see 'Dealing with cupy' section below):
  conda install -c conda-forge cupy cudatoolkit=11.0
  # test installation:
  python -c 'import npyx' # should not return any error
Advanced installation
  • if you want the very latest version:

    conda create -n my_env python=3.10
    conda activate my_env
    pip install git+https://github.com/m-beau/NeuroPyxels@master
    # optionally (see 'Dealing with cupy' section below):
    conda install -c conda-forge cupy cudatoolkit=11.0
    # test installation:
    python -c 'import npyx' # should not return any error
  • If you want to edit npyx locally and eventually contribute:

    💡 Tip: in an ipython/jupyter session, use %load_ext autoreload then %autoreload 2 to make your local edits active in your session without having to restart your kernel. Amazing for development.

    conda create -n my_env python=3.10
    conda activate my_env
    cd path/to/save_dir # any directory where your code will be accessible by your editor and safe. NOT downloads folder.
    git clone https://github.com/m-beau/NeuroPyxels
    cd NeuroPyxels
    pip install . # this will create an egg link to save_dir, which means that you do not need to reinstall the package each time you edit it (e.g. after pulling from github).
    # optionally (see 'Dealing with cupy' section below):
    conda install -c conda-forge cupy cudatoolkit=11.0
    # test installation:
    python -c 'import npyx' # should not return any error

    and pull every now and then:

    cd path/to/save_dir/NeuroPyxels
    git pull
    # And that's it, thanks to the egg link no need to reinstall the package!

Npyx supports Python >=3.7.

Dealing with cupy (GPU shenanigans)

To run some preprocessing functions, you will need NVIDIA drivers and cuda-toolkit installed on your computer. It is a notorious source of bugs. To test your CUDA installation do the following:

nvidia-smi # Should show how much your GPU is being used right now
nvcc # This is the CUDA compiler

If it doesn't work, try up/downgrading the version of cudatoolkit installed:

# check the current version
conda activate my_env
conda list cudatoolkit
# E.g. install version 10.0
conda activate my_env
conda remove cupy, cudatoolkit
conda install -c conda-forge cupy cudatoolkit=10.0

Test installation

You can use the built-in unit testing function 'test_npyx' to make sure that npyx core functions run smoothly, all at once.

from npyx.testing import test_npyx

# any spike sorted recording compatible with phy
# (e.g. kilosort output)
dp = 'datapath/to/myrecording'
test_npyx(dp)

# if any test fails, re-run them with the following to print the error log, and try to fix it or post an issue on github:
test_npyx(dp, raise_error=True)

--- npyx version 2.3.4 unit testing initiated, on directory /media/maxime/AnalysisSSD/test_dataset_artefact...

--- Successfully ran 'read_metadata' from npyx.inout.
--- Successfully ran 'get_npix_sync' from npyx.inout.
--- Successfully ran 'get_units' from npyx.gl.
--- Successfully ran 'ids' from npyx.spk_t.
--- Successfully ran 'trn' from npyx.spk_t.
--- Successfully ran 'trn_filtered' from npyx.spk_t.
--- Successfully ran 'wvf' from npyx.spk_wvf.
--- Successfully ran 'wvf_dsmatch' from npyx.spk_wvf.
--- Successfully ran 'get_peak_chan' from npyx.spk_wvf.
--- Successfully ran 'templates' from npyx.spk_wvf.
--- Successfully ran 'ccg' from npyx.corr.
--- Successfully ran 'plot_wvf' from npyx.plot.
--- Successfully ran 'plot_ccg' from npyx.plot.
--- Successfully ran 'plot_raw' from npyx.plot.

(bunch of plots...)
⚠️ Known installation issues
  • cannot import numba.core hence cannot import npyx
    Older versions of numba did not feature the .core submodule. If you get this error, you are probably running a too old version of numba. Make sure that you have installed npyx in a fresh conda environment if that happens to you. If you still get an error, check that numba is not installed in your root directory.

    pip uninstall numba
    conda activate my_env
    pip uninstall numba
    pip install numba
    

  • core dumped when importing
    This seems to be an issue related to PyQt5 required by opencv (opencv-python). Solution (from post):
# activate npyx environment first
pip uninstall PyQt5
pip uninstall opencv-python
pip install opencv-python
# pip install other missing dependencies

Full log:

In [1]: from npyx import *
In [2]: QObject::moveToThread: Current thread (0x5622e1ea6800) is not the object's thread (0x5622e30e86f0).
Cannot move to target thread (0x5622e1ea6800)

qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/home/maxime/miniconda3/envs/npyx/lib/python3.7/site-packages/cv2/qt/plugins" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb, eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, webgl.

Aborted (core dumped)

  • I think I installed everything properly, but npyx is not found if I run 'python -c "import npyx" '!
    Typically:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'npyx'

Make sure that the python installation that you are using is indeed the version of your new environment.
To do so, in your terminal, run "which python" on linux/mac or "where python" on windows: the output should be the path to the right environment e.g. "/home/.../anaconda/envs/npyx/bin/python". If it isn't, try to deactivate/reactivate your conda environment, or make sure you do not have conflicting python installations on your machine.

🤗 Support and citing

If you find Neuropyxels useful in your work, we kindly request that you cite:

Maxime Beau, Federico D'Agostino, Ago Lajko, Gabriela Martínez, Michael Häusser & Dimitar Kostadinov. NeuroPyxels: loading, processing and plotting Neuropixels data in python. Zenodo https://doi.org/10.5281/zenodo.5509733 (2021).

You can additionally star this repo using the top-right star button to help it gain more visibility.

Cheers!

🔍️ Documentation:

Npyx works with the data formatting employed by SpikeGLX and OpenEphys (binary data and meta data) used in combination with Phy-compatible spike-sorters (Kilosort, SpyKING CIRCUS...). Any dataset compatible with phy can also be analyzed with npyx, in essence.

💡 Design philosophy

  • Memoization (a.k.a. caching)

    Npyx is fast because it rarely computes the same thing twice by relying heavily on caching (memoization as purists like to call it) - in the background, it saves most relevant outputs (spike trains, waveforms, correlograms...) at npix_dataset/npyxMemory, from where they are simply reloaded if called again.

    An important argument controlling this behaviour is again (boolean), by default set to False: if True, most npyx functions will recompute their output rather than loading it from npyxMemory. This is important to be aware of this behaviour, as it can lead to mind boggling bugs. For instance, if you load a spike train then re-curate your dataset, e.g. by splitting unit 56 into 504 and 505, the train of the old 'unit 56' will still exist at kilosort_dataset/npyxMemory and you will remain able to load it even though the unit is gone!

    Under the hood, this caching is implemented in 2 ways: 1) 'home-made' caching inside the Neuropixels data directory itself dp/npyxMemory, 2) joblib-based caching in a hidden directory in the home folder, ./NeuroPyxels. The latter can get quite big quite fast: it is possible to customize its location by simply editing the directory inside npyx/CONFIG.py. If you installed NeuroPyxels as an advanced user, it will be located where you cloned the git repo. If you pip-installed NeuroPyxels, it should be located where your python packages are stored: at ~/anaconda3/envs/npyx/lib/python3.1/site-packages/npyx/ for an installation inside a conda 'npyx' environment, for instance.

  • Ubiquitous arguments

    Most npyx functions take at least one input: dp, which is the path to your Neuropixels-phy dataset. You can find a full description of the structure of such datasets on the phy documentation.

    Other typical parameters are: verbose (whether to print a bunch of informative messages, useful when debugging), saveFig (boolean) and saveDir (whether to save the figure in saveDir for plotting functions).

    Importantly, dp can also be the path to a merged dataset, generated with npyx.merge_datasets() - every function will run as smoothly on merged datasets as on any regular dataset. See below for more details.

  • Minimal and modular reliance of spike-sorter output

    Every function requires the files myrecording.ap.meta/myrecording.oebin (metadata from SpikeGLX/OpenEphys), params.py, spike_times.npy and spike_clusters.npy.

    If you have started spike sorting, cluster_groups.tsv will also be required obviously (will be created filled with 'unsorted' groups if none is found).

    Then, specific functions will require specific files: loading waveforms with npyx.spk_wvf.wvf or extracting your sync channel with npyx.io.get_npix_sync require the raw data myrecording.ap.bin, npyx.spk_wvf.templates the files templates.npy and spike_templates.npy, and so on. This allows you to only transfer the strictly necassary files for your use case from a machine to the next: for instance, if you only want to make behavioural analysis of spike trains but do not care about the waveforms, you can run get_npix_sync on a first machine (which will generate a sync_chan folder containing extracted onsets/offsets from the sync channel(s)), then exclusively transfer the dataset/sync_chan/ folder along with spike_times.npy and spike_clusters.npy (all very light files) on another computer and analyze your data there seemlessly.

📁 Directory structure

The dp parameter of all npyx functions must be the absolute path to myrecording below.

For SpikeGLX recordings:

myrecording/
  myrecording.ap.meta
  params.py
  spike_times.npy
  spike_clusters.npy
  cluster_groups.tsv # optional, if manually curated with phy
  myrecording.ap.bin # optional, if wanna plot waveforms

  # other kilosort/spyking circus outputs here

For Open-Ephys recordings:

myrecording/
  myrecording.oebin
  params.py
  spike_times.npy
  spike_clusters.npy
  cluster_groups.tsv # if manually curated with phy

  # other spikesorter outputs here

  continuous/
    Neuropix-PXI-100.somethingsomething (1, AP...)/
      continuous.dat # optional, if wanna plot waveforms
    Neuropix-PXI-100.somethingsomething (2, LFP...)/
      continuous.dat # optional, if want to plot LFP with plot_raw

  events/
    Neuropix-PXI-100.somethingsomething (1, AP...)/
      TTL somethingelse/
        timestamps.npy # optional, if need to get synchronyzation channel to load with get_npix_sync e.g. to merge datasets
    Neuropix-PXI-100.somethingsomething (2, LFP...)/
      TTL somethingelse/
        timestamps.npy # same timestamps for LFP channel

👉 Common use cases

Load recording metadata

from npyx import *

dp = 'datapath/to/myrecording'

# load contents of .lf.meta and .ap.meta or .oebin files as python dictionnary.
# The metadata of the high and lowpass filtered files are in meta['highpass'] and meta['lowpass']
# Quite handy to get probe version, sampling frequency, recording length etc
meta = read_metadata(dp) # works for spikeGLX (contents of .meta files) and open-ephys (contents of .oebin file)

Load synchronization channel

from npyx.inout import get_npix_sync # star import is sufficient, but I like explicit imports!

# If SpikeGLX: slow the first time, then super fast
onsets, offsets = get_npix_sync(dp, filt_key='highpass') # works for spikeGLX (extracted from .ap.bin file) and open-ephys (/events/..AP/TTL/timestamps.npy)
# onsets/offsets are dictionnaries
# keys: ids of sync channel where a TTL was detected (0,1,2... for spikeGLX, name of TTL folders in events/..AP for openephys),
# values: times of up (onsets) or down (offsets) threshold crosses, in seconds.

Preprocess binary data

Makes a preprocessed copy of the binary file in dp, moves original binary file at dp/original_data This will be as fast as literally copying your file, with a decent GPU!

from npyx.inout import preprocess_binary_file # star import is sufficient, but I like explicit imports!

# can perform bandpass filtering (butterworth 3 nodes) and median subtraction (aka common average referenceing, CAR)
# in the future: ADC realignment (like CatGT), whitening, spatial filtering (experimental).
filtered_fname = preprocess_binary_file(dp, filt_key='ap', median_subtract=True, f_low=None, f_high=300, order=3, verbose=True)

Get good units from dataset

from npyx.gl import get_units
units = get_units(dp, quality='good')

Load spike times from unit u

from npyx.spk_t import trn
u=234
t = trn(dp, u) # gets all spikes from unit 234, in samples

Load waveforms from unit u

from npyx.inout import read_spikeglx_meta
from npyx.spk_t import ids, trn
from npyx.spk_wvf import get_peak_chan, wvf, templates

# returns a random sample of 100 waveforms from unit 234, in uV, across 384 channels
waveforms = wvf(dp, u) # return array of shape (n_waves, n_samples, n_channels)=(100, 82, 384) by default
waveforms = wvf(dp, u, n_waveforms=1000, t_waveforms=90) # now 1000 random waveforms, 90 samples=3ms long

# Get the unit peak channel (channel with the biggest amplitude)
peak_chan = get_peak_chan(dp,u)
# extract the waveforms located on peak channel
w=waves[:,:,peak_chan]

# Extract waveforms of spikes occurring between
# 0-100s and 300-400s in the recording,
# because that's when your mouse sneezed
waveforms = wvf(dp, u, periods=[(0,100),(300,400)])

# alternatively, longer but more flexible:
fs=meta['highpass']['sampling_rate']
t=trn(dp,u)/fs # convert in s
# get ids of unit u: all spikes have a unique index in the dataset,
# which is their rank sorted by time (as in spike_times.npy)
u_ids = ids(dp,u)
ids=ids(dp,u)[(t>900)&(t<1000)]
mask = (t<100)|((t>300)&(t<400))
waves = wvf(dp, u, spike_ids=u_ids[mask])

# If you want to load the templates instead (faster and does not require binary file):
temp = templates(dp,u) # return array of shape (n_templates, 82, n_channels)

Compute auto/crosscorrelogram between 2 units

from npyx.corr import ccg, ccg_stack

# returns ccg between 234 and 92 with a binsize of 0.2 and a window of 80
c = ccg(dp, [234,92], cbin=0.2, cwin=80)

# Only using spikes from the first and third minutes of recording
c = ccg(dp, [234,92], cbin=0.2, cwin=80, periods=[(0,60), (120,180)])

# better, compute a big stack of crosscorrelograms with a given name
# The first time, CCGs will be computed in parallel using all the available CPU cores
# and it will be saved in the background and, reloadable instantaneously in the future
source_units = [1,2,3,4,5]
target_units = [6,7,8,9,10]
c_stack = ccg_stack(dp, source_units, target_units, 0.2, 80, name='my_relevant_ccg_stack')
c_stack = ccg_stack(dp, name='my_relevant_ccg_stack') # will work to reaload in the future

Plot waveform and crosscorrelogram of unit u

# all plotting functions return matplotlib figures
from npyx.plot import plot_wvf, get_peak_chan

u=234
# plot waveform, 2.8ms around templates center, on 16 channels around peak channel
# (the peak channel is found automatically, no need to worry about finding it)
fig = plot_wvf(dp, u, Nchannels=16, t_waveforms=2.8)

# But if you wished to get it, simply run
peakchannel = get_peak_chan(dp, u)

# plot ccg between 234 and 92
# as_grid also plot the autocorrelograms
fig = plot_ccg(dp, [u,92], cbin=0.2, cwin=80, as_grid=True)

Preprocess your waveforms (drift-shift-matching) and spike trains (detect periods with few false positive/negative)

# all plotting functions return matplotlib figures
from npyx.spk_wvf import wvf_dsmatch
from npyx.spk_t import trn_filtered

# wvf_dsmatch subselect 'best looking' waveforms
# by first matching them by drift state (Z, peak channel and XY, amplitude on peak channel)
# then shifting them around to realign them (using the crosscorr of its whole spatial footprint)
# on the plot, black is the original waveform as it would be plotted in phy,
# green is drift-matched, red is drift-shift matched
w_preprocessed = wvf_dsmatch(dp, u, plot_debug=True)

# trn_filtered clips the recording in 10s (default) chunks
# and estimates the false positive/false negative spike sporting rates on such chunks
# before masking out spikes occurring inside 'bad chunks',
# defined as chunks with too high FP OR FN rates (5% and 5% by default)
t_preprocessed = trn_filtered(dp, u, plot_debug=True)

Plot chunk of raw data with overlaid units

units = [1,2,3,4,5,6]
channels = np.arange(70,250)
# raw data are whitened, high-pass filtered and median-subtracted by default - parameters are explicit below
plot_raw_units(dp, times=[0,0.130], units = units, channels = channels,
               colors=['orange', 'red', 'limegreen', 'darkgreen', 'cyan', 'navy'],
               lw=1.5, offset=450, figsize=(6,16), Nchan_plot=10,
               med_sub=1, whiten=1, hpfilt=1)

Plot peri-stimulus time histograms across neurons and conditions

# Explore responses of 3 neurons to 4 categories of events:
fs=30000 # Hz
units=[1,2,3]
trains=[trn(dp,u)/fs for u in units] # make list of trains of 3 units
trains_str=units # can give specific names to units here, show on the left of each row
events=[licks, sneezes, visual_stimuli, auditory_stimuli] # get events corresponding to 4 conditions
events_str=['licking', 'sneezing', 'visual_stim', 'auditory_stim'] # can give specific names to events here, show above each column
events_col='batlow' # colormap from which the event colors will be drawn
fig=summary_psth(trains, trains_str, events, events_str, psthb=10, psthw=[-750,750],
                 zscore=0, bsl_subtract=False, bsl_window=[-3000,-750], convolve=True, gsd=2,
                 events_toplot=[0], events_col=events_col, trains_col_groups=trains_col_groups,
                 title=None, saveFig=0, saveDir='~/Downloads', _format='pdf',
                 figh=None, figratio=None, transpose=1,
                 as_heatmap=False,  vmin=None, center=None, vmax=None, cmap_str=None)

Merge datasets acquired on two probes simultaneously

# The three recordings need to include the same sync channel.
from npyx.merger import merge_datasets
dps = ['same_folder/lateralprobe_dataset',
       'same_folder/medialprobe_dataset',
       'same_folder/anteriorprobe_dataset']
probenames = ['lateral','medial','anterior']
dp_dict = {p:dp for p, dp in zip(dps, probenames)}

# This will merge the 3 datasets (only relevant information, not the raw data) in a new folder at
# dp_merged: same_folder/merged_lateralprobe_dataset_medialprobe_dataset_anteriorprobe_dataset
# where all npyx functions can smoothly run.
# The only difference is that units now need to be called as floats,
# of format u.x (u=unit id, x=dataset id [0-2]).
# lateralprobe, medial probe and anteriorprobe x will be respectively 0,1 and 2.
dp_merged, datasets_table = merge_datasets(dp_dic)


--- Merged data (from 2 dataset(s)) will be saved here: /same_folder/merged_lateralprobe_dataset_medialprobe_dataset_anteriorprobe_dataset.

--- Loading spike trains of 2 datasets...

sync channel extraction directory found: /same_folder/lateralprobe_dataset/sync_chan
Data found on sync channels:
chan 2 (201 events).
chan 4 (16 events).
chan 5 (175 events).
chan 6 (28447 events).
chan 7 (93609 events).
Which channel shall be used to synchronize probes? >>> 7

sync channel extraction directory found: /same_folder/medialprobe_dataset/sync_chan
Data found on sync channels:
chan 2 (201 events).
chan 4 (16 events).
chan 5 (175 events).
chan 6 (28447 events).
chan 7 (93609 events).
Which channel shall be used to synchronize probes? >>> 7

sync channel extraction directory found: /same_folder/anteriorprobe_dataset/sync_chan
Data found on sync channels:
chan 2 (201 events).
chan 4 (16 events).
chan 5 (175 events).
chan 6 (28194 events).
chan 7 (93609 events).
Which channel shall be used to synchronize probes? >>> 7

--- Aligning spike trains of 2 datasets...
More than 50 sync signals found - for performance reasons, sub-sampling to 50 homogenoeously spaced sync signals to align data.
50 sync events used for alignement - start-end drift of -3080.633ms

--- Merged spike_times and spike_clusters saved at /same_folder/merged_lateralprobe_dataset_medialprobe_dataset_anteriorprobe_dataset.

--> Merge successful! Use a float u.x in any npyx function to call unit u from dataset x:
- u.0 for dataset lateralprobe_dataset,
- u.1 for dataset medialprobe_dataset,
- u.2 for dataset anteriorprobe_dataset.

Now any npyx function runs on the merged dataset! Under the hood, it will create a merged_dataset_dataset1_dataset2/npyxMemory folder to save any data computed across dataframes, but will use the original dataset1/npyxMemory folder to save data related to this dataset exclusively (e.g. waveforms). Hence, there is no redundancy: space and time are saved.

This is also why it is primordial that you do not move your datatasets from their original paths after merging them - else, functions ran on merged_dataset1_dataset2 will not know where to go fetch the data! They refer to the paths in merged_dataset_dataset1_dataset2/datasets_table.csv. If you really need to, you can move your datasets but do not forget to edit this file accordingly.

# These will work!
t = trn(dp_merged, 92.1) # get spikes of unit 92 in dataset 1 i.e. medialprobe
fig=plot_ccg(dp_merged,[10.0, 92.1, cbin=0.2, cwin=80]) # compute CCG between 2 units across datasets

PS - The spike times are aligned across datasets by modelling the drift between the clocks of the neuropixels headstages linearly: TTL probe 1 = a * TTL probe 1 + b (if a!=1, there is drift between the clocks), so spiketimes_probe2_aligned_to_probe1 = a * spiketimes_probe2 + b

⭐ Bonus: matplotlib plot prettifier

from npyx.plot import get_ncolors_cmap

# allows you to easily extract the (r,g,b) tuples from a matplotlib or crameri colormap
# to use them in other plots!
colors = get_ncolors_cmap('coolwarm', 10, plot=1)
colors = get_ncolors_cmap('viridis', 10, plot=1)
# in a jupyter notebook, will also plot he HTML colormap:

from npyx.plot import mplp
import matplotlib.pyplot as plt

# mplp() will turn any matplotlib plot into something you can work with.
# fed up googling around and landing on stack overflow to tweak your figures?
# just read mplp parameters, they are self-explanatory!

df1 = pd.load("my_dataframe.csv")

# Seaborn figure (seaborn is simply a wrapper for matplotlib):
fig = plt.figure()
sns.scatterplot(data=df1,
                x='popsync', y='depth', hue='mean_popsync',
                palette='plasma', alpha=1, linewidth=1, edgecolor='black')

# Same figure, tweaked with mplp():
fig = plt.figure()
sns.scatterplot(data=df1,
                x='popsync', y='depth', hue='mean_popsync',
                palette='plasma', alpha=1, linewidth=1, edgecolor='black')
mplp(figsize=(3,3), title="My title", ylim=[-10,-2], xlim=[-40,60],
      xlabel = "My x label (rotated ticks)", ylabel="My y label",
      xtickrot=45,
      hide_legend=True, colorbar=True,
      vmin=df['mean_popsync'].min(), vmax=df['mean_popsync'].max(),
      cbar_w=0.03, cbar_h=0.4, clabel="My colorbar label\n(no more ugly legend!)", cmap="plasma",
      clabel_s=16, cticks_s=14, ticklab_s=16,
      saveFig=saveFig, saveDir=saveDir, figname = f"popsync_{pair}")


neuropyxels's People

Contributors

agolajko avatar brandonstell avatar chris-angeloni avatar fededagos avatar francisconaveros avatar herzfeldd avatar julie-fabre avatar m-beau avatar mgmartinezl avatar neuromantic99 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

neuropyxels's Issues

The size of .Neuropyxels folder causes disc size problems on the server

Hi Maxime, for C4 version of npyx, we are having problems with expanding .Neuropixels folder for each user on the server. It is growing up to 500 GB for a recording of 90 GB despite deleting that folder between recordings. There could be two solutions I can think of: Either we should be able to direct this folder under other drives - not under /home directory or nypx can delete it just after it finishes processing unless it needs it to reopen the same recording.

Can you help us solve this issue? We're running into the same issue multiple times in a week.

Thanks
Sevgi - Hull lab

get sync channel bug

get_npix_sync is always going to look for an offset file, even when the signal went up and never down, leading to the function crashing:

Sync channel extraction directory found: E:\NP_data_temp\210805_xg020_01\210805_xg020_01_g0\sync_chan

Sync channel onsets extracted from highpass (ap) file found and loaded.
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-29-e144e38dfad0> in <module>
      1 # sync channel
----> 2 onsets, offsets = get_npix_sync(dp)
      3 camera_frames = onsets[6]

D:\anaconda\envs\npyx\lib\site-packages\npyx\io.py in get_npix_sync(dp, output_binary, filt_key, unit)
    382                         file_i = ale(file[-15])
    383                         onsets[file_i]=np.load(sync_dp/file)/srate
--> 384                         offsets[file_i]=np.load(sync_dp/(file[:-13]+'f'+file[-12:]))/srate
    385             if any(onsets):
    386                 # else, might be that sync_dp is empty

D:\anaconda\envs\npyx\lib\site-packages\numpy\lib\npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
    415             own_fid = False
    416         else:
--> 417             fid = stack.enter_context(open(os_fspath(file), "rb"))
    418             own_fid = True
    419 

FileNotFoundError: [Errno 2] No such file or directory: 'E:\\NP_data_temp\\210805_xg020_01\\210805_xg020_01_g0\\sync_chan\\210805_xg020_01_g0.imec0.ap_sync3of_samples.npy'

preprocess lf data?

Is it possible to use preprocess_binary_file to preprocess lf files? I attempted it (below, and also with f_high=None). Is this possible and I am not calling it correctly, or is it impossible currently?

filtered_fname = preprocess_binary_file(dp, filt_key='lf', fname=None, target_dp=None, move_orig_data=True, median_subtract=False, f_low=250, f_high=0, order=3, verbose=True)
Preprocessing D:\MarieQD\LFP\1657_220729_g1_t0.imec0.lf.bin...
- filtering in time (between 250 and 0 Hz) forward:True, backward:False.
Traceback (most recent call last):
File "", line 1, in
File "d:\marieqd\npyx\neuropyxels\npyx\inout.py", line 760, in preprocess_binary_file
meta = read_metadata(dp)
File "d:\marieqd\npyx\neuropyxels\npyx\inout.py", line 47, in read_metadata
meta = metadata(dp)
File "d:\marieqd\npyx\neuropyxels\npyx\inout.py", line 199, in metadata
if 'imProbeOpt' in meta_glx["highpass"].keys(): # 3A
KeyError: 'highpass'

Issue with the outputs of get_npix_sync

Hi!

I noticed somethign with the outputs of npyx.inout.get_npix_sync. If calculating the the onset (or offset) times using some filt_key value (e.g. filt_key = 'highpass') everything works quite well on the first pass. As I understand it, if I wanna retrieve that information a second time, the function tries to find if this was already calculated from before and loads it from the file it created on the first run (in the folder sync_chan)

Now let's say I try to run the function with filt_key = 'lowpass' on the second time (using 'highpass' the first). On the second run, the onset times come out wrong, most likely because the function calculates them using the sampling rate previously saved (taken from the AP band used in the first run with filt_key = 'highpass'). The analysis is not interchangeable between filt_key values, and would return proper onset or offset times only if always using the same filt_key stored int the files it creates. Otherwise it converts from samples to time using a wrong sampling rate.

Error in "dealing with cupy"

I'm having some trouble with the installation. When I run the first part of the code (ending with " python -c 'import npyx'") I don't seem to get any errors, just this warning: "UserWarning: CUDA path could not be detected. Set CUDA_PATH environment variable if CuPy fails to load."

When I move on to test the installation, "nvidia-smi" seems to work fine, but "nvcc" returns this error:
"nvcc : The term 'nvcc' is not recognized as the name of a cmdlet, function, script file, or operable program. Check
the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1

  • nvcc
  •   + CategoryInfo          : ObjectNotFound: (nvcc:String) [], CommandNotFoundException
      + FullyQualifiedErrorId : CommandNotFoundException"
    
    

How do I resolve this?
I'm quite new to coding, so I apologize if the answer is obvious.

Partially corrupted binary file handling in wvf_dsmatch

It can occur that some portions of binary files are, for some reason, corrupted (impossible to read 1 second of data in a file for instance but everything else looks fine...)

This issue was handled by skipping waveforms in such problematics sections:

print(f"WARNING it seems the binary file at {dp} is corrupted. Waveform {i} (at byte {t1}, {t1/n_channels_dat/item_size/sample_rate}s) could not be loaded.")

But this fix currently isn't compatible with dsmatch, causing functions to crash at reshape time:

raw_waves = raw_waves.reshape(spike_ids_split.shape[0], n_waveforms_per_batch, t_waveforms, -1)

Adjust the expected 'n' waveforms based on the amount of dropped waveforms in wvf. Requires to know which ones to make sure ids are not scrambled, and probably to drop the full waveform batch around dropped waveform to maintain shape integrity.

Very niche edge case - note for the future in case a user experiences this issue.

TypeError from get_npix_sync(dp)

Hello, I'd love to try using NeuroPyxels, but when I attempt to run onsets, offsets = get_npix_sync(dp), I get the following error:


---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[28], line 4
      1 from npyx.inout import get_npix_sync # star import is sufficient, but I like explicit imports!
      3 # If SpikeGLX: slow the first time, then super fast
----> 4 onsets, offsets = get_npix_sync(dp)
      5 # keys: ids of sync channel where a TTL was detected,
      6 # values: times of up (onsets) or down (offsets) threshold crosses, in seconds.

File ~/opt/anaconda3/envs/npix/lib/python3.10/site-packages/npyx/inout.py:461, in get_npix_sync(dp, output_binary, filt_key, unit, verbose, again)
    458 srate = meta[filt_key]['sampling_rate'] if unit=='seconds' else 1
    460 if meta['acquisition_software']=='OpenEphys':
--> 461     raise('OpenEphys sync channel loading not implemented yet - manually load ./Neuropix-PXI-100.0/TTL_1/timestamps.npy.')
    462     filt_id = 0 if filt_key=='highpass' else 1
    463     timestamps = np.load(dp/f'events/Neuropix-PXI-100.{filt_id}/TTL_1/timestamps.npy')

TypeError: exceptions must derive from BaseException

Any suggestions?

Thank you!

Tensorflow in setup.py

Tensorflow is part of setup.py but it's not used anywhere in the codebase. Can we remove it please? It makes the loading times slower and means there will be extra stuff downloaded for users

Sampling rate from metadata converted to int

Hi all,
First of all, thanks for the wonderful package!
I was fiddling around with loading some metadata from a recording and realised that the sampling rate returned from the npyx.inout.metadata function is converted to int (line 262 in the code):

            if meta_glx[filt_key]['typeThis'] == 'imec':
                meta[filt_key]['sampling_rate']=int(meta_glx[filt_key]['imSampRate'])

If, by some circumstances, one has more than one event, wouldn't this lead to accumulated error? The calibrated sampling rate from my recording is 30000.352941 (which is the sampling rate .meta file), but from the metadata function I'd get 30000.

Ultimately, I worry that it might make event detection (e.g. from the sync channel) a little bit inaccurate (although the error would be very small, almost negligible).

4 shanks channel map waveform plotting bug

fig = plot_wvf(dp, 4, Nchannels=16, t_waveforms=2.8)

Automatic pdb calling has been turned ON
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-49-ef61c94e617f> in <module>
      1 get_ipython().run_line_magic('pdb', 'ON')
----> 2 fig = plot_wvf(dp, 4, Nchannels=16, t_waveforms=2.8)

D:\anaconda\envs\npyx\lib\site-packages\npyx\plot.py in plot_wvf(dp, u, Nchannels, chStart, n_waveforms, t_waveforms, periods, spike_ids, wvf_batch_size, ignore_nwvf, again, whiten, med_sub, hpfilt, hpfiltf, nRangeWhiten, nRangeMedSub, title, plot_std, plot_mean, plot_templates, color, labels, scalebar_w, ticks_lw, sample_lines, ylim, saveDir, saveFig, saveData, _format, ignore_ks_chanfilt, ax_edge_um_x, ax_edge_um_y, margin, figw_inch, figh_inch, as_heatmap)
    571                   memorysafe=False, fast = False )[1]
    572     assert waveforms.shape[0]!=0,'No waveforms were found in the provided periods!'
--> 573     assert waveforms.shape[1:]==(t_waveforms_s, cm.shape[0])
    574     tplts=templates(dp, u, ignore_ks_chanfilt=ignore_ks_chanfilt)
    575     assert tplts.shape[2]==waveforms.shape[2]==cm.shape[0]

AssertionError: 

> d:\anaconda\envs\npyx\lib\site-packages\npyx\plot.py(573)plot_wvf()
    571                   memorysafe=False, fast = False )[1]
    572     assert waveforms.shape[0]!=0,'No waveforms were found in the provided periods!'
--> 573     assert waveforms.shape[1:]==(t_waveforms_s, cm.shape[0])
    574     tplts=templates(dp, u, ignore_ks_chanfilt=ignore_ks_chanfilt)
    575     assert tplts.shape[2]==waveforms.shape[2]==cm.shape[0]

ipdb> cm.shape
(377, 3)
ipdb> waveforms.shape[1:]
(84, 384)
ipdb> (t_waveforms_s, cm.shape[0])
(84, 377)

waveform width sanity check bug

recording time: 26684033530/30000/385/2

Invalid times: [ 4074484  4077814  4081952  4083699  4083839  4088920  4090476  4091892
  4094331  4094451 11120999 11122511 11124662 11127079 11127632 11128641
 11131807 11133788 11134566 11134731 14914721 14915521 14916579 14920223
 14921298 14921539 14921967 14923478 14925331 14925980 27688802 27689220
 27689426 27691673 27696618 27698414 27700627 27703501 27705464 27705596
 31941074 31954800 31956367 31957481 31958612 31963028 31963189 31964844
 31965333 31969095]

Can't read Neuropixels Ultra SpikeGLX metadata

read_metadata fails with a Neuropixels Ultra SpikeGLX metadata file. I first get "WARNING probe version 1123.0 not handled - post an issue at www.github.com/m-beau/NeuroPyxels", and then I get the following exception:

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\u239257\Anaconda3\envs\phyllum_cell_type_classifier\lib\site-packages\npyx\inout.py", line 48, in read_metadata
meta = metadata(dp)
File "C:\Users\u239257\Anaconda3\envs\phyllum_cell_type_classifier\lib\site-packages\npyx\inout.py", line 241, in metadata
meta['bit_uV_conv_factor']=(Vrange/2**bits_encoding/ampFactor)
UnboundLocalError: local variable 'bits_encoding' referenced before assignment

The variable bits_encoding is not set because the probe version isn't one of the four recognized ones. I'm using the cell_type_classifier branch, but as far as I can tell, the relevant code seems identical in the master and m-beau branches.

H5 Dump Requires Sync/Stim Channel

The current H5 code assumes the presence of an opto stim channel (and errors in get_stim_chan if no sync channel exists. This should allow H5 dumps without a stim channel - in which case, sane_spikes should include all spikes.

error with preprocess_binary_file

I'm getting this error when I run preprocess_binary_file after the preprocessing is run:

can't invoke "event" command: application has been destroyed
while executing
"event generate $w <>"
(procedure "ttk::ThemeChanged" line 6)
invoked from within
"ttk::ThemeChanged"

Not sure what this is or if it is important.

This is the exact command

filtered_fname = preprocess_binary_file(dp, filt_key='ap', fname=None, target_dp=None, move_orig_data=True, median_subtract=True, f_low=None, f_high=300, order=3, verbose=True)

Only seems to happen sometimes! Thanks.

Y-axis range adjustment for psth plot with raster

Hi, when plotting the Raster and PSTH plot, it would be useful to be able to manually set the y-axis range (ylim) for the bottom psth plot. This is so that they y-range can be shared when comparing to units or the same units during different conditions

Permission error on ubuntu VM

I am installing Neuropyxels on a ubuntu VM on our windows server. I am getting this permission error when I try to run the filtering algorithm:

iltered_fname = preprocess_binary_file(dp, filt_key='ap', fname=None, target_dp=None, move_orig_data=False, median_subtract=True, f_low=None, f_high=300, order=3, verbose=True)
Preprocessing /home/[email protected]/Ddrive/MarieQD/DH12_211201g2/DH12_21_12_01_g2_t0.imec0.ap.bin...
- median subtraction (aka common average referencing CAR),
- filtering in time (between 0 and 300 Hz) forward:True, backward:False.
Traceback (most recent call last):
File "", line 1, in
File "/home/[email protected]/NeuroPyxels/npyx/inout.py", line 812, in preprocess_binary_file
with open(target_dp / filtered_fname, 'wb') as fw: # open for writing processed data
PermissionError: [Errno 13] Permission denied: '/home/[email protected]/Ddrive/MarieQD/DH12_211201g2/DH12_21_12_01_g2_t0.imec0_medsub_tempfiltNone300TrueFalse.ap.bin'

Any help appreciated! We ran into a very niche permission error when I was running this on the windows side, but because of GPU issues we had to switch to this ubuntu VM.

IndexError in funtion trn_filtered

When I execute the command predict_cell_types, I´m having the next error:

Traceback (most recent call last):
File "C:\Users\Usuario.conda\envs\phy_dev\lib\site-packages\joblib\externals\loky\process_executor.py", line 428, in _process_worker
r = call_item()
File "C:\Users\Usuario.conda\envs\phy_dev\lib\site-packages\joblib\externals\loky\process_executor.py", line 275, in call
return self.fn(*self.args, **self.kwargs)
File "C:\Users\Usuario.conda\envs\phy_dev\lib\site-packages\joblib_parallel_backends.py", line 620, in call
return self.func(*args, **kwargs)
File "C:\Users\Usuario.conda\envs\phy_dev\lib\site-packages\joblib\parallel.py", line 288, in call
return [func(*args, **kwargs)
File "C:\Users\Usuario.conda\envs\phy_dev\lib\site-packages\joblib\parallel.py", line 288, in
return [func(*args, **kwargs)
File "c:\phyllum\neuropyxels-master\npyx\c4\predict_cell_types.py", line 237, in aux_prepare_dataset
t, _ = trn_filtered(dp, u, period_m=None, again=True, enforced_rp=-1)
File "c:\phyllum\neuropyxels-master\npyx\spk_t.py", line 772, in trn_filtered
good_spikes_m, good_fp_start_end, good_fn_start_end = train_quality(dp, unit, period_m,
File "c:\phyllum\neuropyxels-master\npyx\spk_t.py", line 566, in train_quality
unit_amp = load_amplitudes(dp, unit, verbose, 'all', again, enforced_rp)
File "c:\phyllum\neuropyxels-master\npyx\spk_t.py", line 112, in load_amplitudes
unit_ids = ids(dp, unit, True, verbose, periods, again, enforced_rp)
File "c:\phyllum\neuropyxels-master\npyx\spk_t.py", line 92, in ids
indices = indices[~duplicates_m]
IndexError: boolean index did not match indexed array along dimension 0; dimension is 299 but corresponding boolean dimension is 1117
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\Usuario.conda\envs\phy_dev\Scripts\predict_cell_types-script.py", line 33, in
sys.exit(load_entry_point('npyx', 'console_scripts', 'predict_cell_types')())
File "c:\phyllum\neuropyxels-master\npyx\c4\predict_cell_types.py", line 513, in main
prediction_dataset, good_units = prepare_dataset(args)
File "c:\phyllum\neuropyxels-master\npyx\c4\predict_cell_types.py", line 316, in prepare_dataset
prediction_dataset, bad_units = prepare_dataset_from_binary_parallel(
File "c:\phyllum\neuropyxels-master\npyx\c4\predict_cell_types.py", line 263, in prepare_dataset_from_binary_parallel
dataset_results = Parallel(n_jobs=num_cores, prefer="processes")(
File "C:\Users\Usuario.conda\envs\phy_dev\lib\site-packages\joblib\parallel.py", line 1098, in call
self.retrieve()
File "C:\Users\Usuario.conda\envs\phy_dev\lib\site-packages\joblib\parallel.py", line 975, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "C:\Users\Usuario.conda\envs\phy_dev\lib\site-packages\joblib_parallel_backends.py", line 567, in wrap_future_result
return future.result(timeout=timeout)
File "C:\Users\Usuario.conda\envs\phy_dev\lib\concurrent\futures_base.py", line 458, in result
return self.__get_result()
File "C:\Users\Usuario.conda\envs\phy_dev\lib\concurrent\futures_base.py", line 403, in __get_result
raise self._exception
IndexError: boolean index did not match indexed array along dimension 0; dimension is 299 but corresponding boolean dimension is 1117

Can't read Neuropixels NHP (commercial release) metadata

Seems to be the exact problem as #378 but now with, WARNING probe version 1030.0 not handled. AP bin .meta file is attached with .txt extension (since GitHub doesn't allow a .meta extension; it's saved as ASCII so I would try to read this file directly or it will throw a codec error). Tiberius_TF_PMD_0425_bank0_1_g0_t0.imec0.ap.meta.txt

SpikeGLX ver.: 20230411, Imex Phase30, v.3.60
Probe Type: NP 1032

Full traceback is as follows,

UnboundLocalError Traceback (most recent call last)
Cell In[15], line 3
1 ap_bin_dir = "/Volumes/Kenji3/Tiberius_TF_PMD_0425_bank0_1_g0/Tiberius_TF_PMD_0425_bank0_1_g0_imec0/"
----> 3 meta = npyx.read_metadata(ap_bin_dir)

File ~/Library/Caches/pypoetry/virtualenvs/cfd-dlpfc-z1Y6YtZg-py3.9/lib/python3.9/site-packages/npyx/inout.py:48, in read_metadata(dp)
46 meta[probe] = metadata(dpx)
47 else:
---> 48 meta = metadata(dp)
50 return meta

File ~/Library/Caches/pypoetry/virtualenvs/cfd-dlpfc-z1Y6YtZg-py3.9/lib/python3.9/site-packages/npyx/inout.py:244, in metadata(dp)
242 bits_encoding=14
243 ampFactor=80 # hardcoded
--> 244 meta['bit_uV_conv_factor']=(Vrange/2**bits_encoding/ampFactor)
247 # find everything else
248 for filt_key in ['highpass','lowpass']:

UnboundLocalError: local variable 'bits_encoding' referenced before assignment

Thank you for making such a useful package!
Kenji

psth plot title

In the psth_plot function, the title is currently set to None, which does not allow to set your own title.

Also, it seems as if you cannot use this function to plot in a loop, see screenshot below.

In  831 for unit in  132, 141, 225

title=', color='darkgreen', figsize=None,

plot_acg function issue

Hi everyone,

this error message came up during my recent analysis, using Maxime's plot_acg() function.
For other datasets and until unit 456 it runs smoothly.
Any suggestions what I could do?

Cheers,
Marie

Screenshot 2021-03-18 at 14 06 55

How to change cache location?

By default the cache location is in my home folder (User/chris) on the C: drive. However, as my cache grows as I analyze more datasets, it is filling my C: drive. Is there a way to modify the cache location to a larger drive?

Error when explore functional connectivity between neurons

Thank you so much for the great tool!

When I tried the NeuroPyxels quickstart.ipnyb, I have some errors:

2.4 - Easily explore functional connectivity between neurons

CCGs parameters

cbin=0.5
cwin=100
n_consec_bins=3
corr_type='inhibitions' #'synchrony'

significance test parameters

p_th=0.01
fract_baseline=4./5
W_sd=10
test='Poisson_Stark'

mfr_th = 40

units=[u for u in get_units(dp) if mfr(dp,u)>mfr_th]
print(f'Found {len(units)} units above {mfr_th}Hz.')

fig=plot_sfcm(dp, corr_type=corr_type, metric='amp_z', cbin=cbin, cwin=cwin,
p_th=p_th, n_consec_bins=n_consec_bins, fract_baseline=fract_baseline, W_sd=W_sd, test=test,
drop_seq=['sign', 'time', 'max_amplitude'],
units=units, name=f'all_units_move_{mfr_th}Hz',
text=False, markers=False, ticks=True, depth_ticks=False,
regions={},reg_colors={},
vminmax=[-7,7], figsize=(4,4),
saveFig=False, saveDir=dp, again=False, againCCG=0, use_template_for_peakchan=True,
periods='all')


Error:

Looking for significant CCGs over 16 cores: 0%| | 0/136 [00:00<?, ?it/s]

_RemoteTraceback Traceback (most recent call last)
_RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Users\muhang\miniconda3\envs\npyx\lib\site-packages\joblib\externals\loky\process_executor.py", line 463, in _process_worker
r = call_item()
File "C:\Users\muhang\miniconda3\envs\npyx\lib\site-packages\joblib\externals\loky\process_executor.py", line 291, in call
return self.fn(*self.args, **self.kwargs)
File "C:\Users\muhang\miniconda3\envs\npyx\lib\site-packages\joblib\parallel.py", line 598, in call
return [func(args, **kwargs)
File "C:\Users\muhang\miniconda3\envs\npyx\lib\site-packages\joblib\parallel.py", line 598, in
return [func(args, **kwargs)
File "D:\repos\NeuroPyxels\npyx\corr.py", line 1258, in get_ccg_sig
crosses+=StarkAbeles2009_ccg_significance(CCG, cbin, p_th, n_consec_bins, 1, W_sd, ret_values=True, only_max=False)
File "D:\repos\NeuroPyxels\npyx\corr.py", line 1162, in StarkAbeles2009_ccg_significance
pred, pvals = StarkAbeles2009_ccg_sig(CCG, W=2
W_sd, WINTYPE='gauss', HF=None, CALCP=True, sgn=sgn)
File "D:\repos\NeuroPyxels\npyx\corr.py", line 1115, in StarkAbeles2009_ccg_sig
winlist[2]:{0: (sgnl.triang(2
W+1), W), # triang even W
AttributeError: module 'scipy.signal' has no attribute 'triang'
"""

The above exception was the direct cause of the following exception:

AttributeError Traceback (most recent call last)
Cell In[167], line 18
15 units=[u for u in get_units(dp) if mfr(dp,u)>mfr_th]
16 print(f'Found {len(units)} units above {mfr_th}Hz.')
---> 18 fig=plot_sfcm(dp, corr_type=corr_type, metric='amp_z', cbin=cbin, cwin=cwin,
19 p_th=p_th, n_consec_bins=n_consec_bins, fract_baseline=fract_baseline, W_sd=W_sd, test=test,
20 drop_seq=['sign', 'time', 'max_amplitude'],
21 units=units, name=f'all_units_move_{mfr_th}Hz',
22 text=False, markers=False, ticks=True, depth_ticks=False,
23 regions={},reg_colors={},
24 vminmax=[-7,7], figsize=(4,4),
25 saveFig=False, saveDir=dp, again=False, againCCG=0, use_template_for_peakchan=True,
26 periods='all')

File D:\repos\NeuroPyxels\npyx\plot.py:2799, in plot_sfcm(dp, corr_type, metric, cbin, cwin, p_th, n_consec_bins, fract_baseline, W_sd, test, drop_seq, units, name, text, markers, ticks, depth_ticks, regions, reg_colors, vminmax, figsize, saveFig, saveDir, _format, again, againCCG, use_template_for_peakchan, periods)
2781 def plot_sfcm(dp, corr_type='connections', metric='amp_z', cbin=0.5, cwin=100,
2782 p_th=0.02, n_consec_bins=3, fract_baseline=4./5, W_sd=10, test='Poisson_Stark',
2783 drop_seq=['sign', 'time', 'max_amplitude'], units=None, name=None,
(...)
2786 saveFig=False, saveDir=None, _format='pdf',
2787 again=False, againCCG=False, use_template_for_peakchan=False, periods='all'):
2788 f'''
2789 Visually represents the connectivity matrix sfcm computed with npyx.corr.gen_sfc().
2790 Each line/row is a unit, sorted by depth, and the colormap corresponds to the 'metric' parameter.
(...)
2796 - fig: matplotlib figure
2797 '''
-> 2799 sfc, sfcm, peakChs, sigstack, sigustack = gen_sfc(dp, corr_type, metric, cbin, cwin,
2800 p_th, n_consec_bins, fract_baseline, W_sd, test,
2801 again, againCCG, drop_seq, None, None, units=units, name=name,
2802 use_template_for_peakchan=use_template_for_peakchan,
2803 periods=periods)
2804 gu = peakChs[:,0]
2805 ch = peakChs[:,1].astype(np.int64)

File D:\repos\NeuroPyxels\npyx\corr.py:1507, in gen_sfc(dp, corr_type, metric, cbin, cwin, p_th, n_consec_bins, fract_baseline, W_sd, test, again, againCCG, drop_seq, pre_chanrange, post_chanrange, units, name, use_template_for_peakchan, periods)
1504 peakChs = npyx.spk_wvf.get_depthSort_peakChans(dp, quality='good', use_template=use_template_for_peakchan)
1505 gu = peakChs[:,0]
-> 1507 sigstack, sigustack, sfc = ccg_sig_stack(dp, gu, gu, cbin, cwin, name,
1508 p_th, n_consec_bins, sgn, fract_baseline, W_sd, test, again, againCCG, ret_features=True, only_max=only_max,
1509 periods=periods)
1512 sfc['t_ms_center'] = sfc.l_ms+(sfc.r_ms-sfc.l_ms)/2
1514 # If filtering of connections wishes to be done at a later stage, simply return

File D:\repos\NeuroPyxels\npyx\corr.py:1373, in ccg_sig_stack(dp, U_src, U_trg, cbin, cwin, name, p_th, n_consec_bins, sgn, fract_baseline, W_sd, test, again, againCCG, ret_features, only_max, periods)
1370 ccgsig_ids.append((i,j))
1371 ccgsig_args.append((CCG, cbin, cwin, p_th, n_consec_bins, sgn, fract_baseline, W_sd, test, ret_features, only_max))
-> 1373 ccgsig_results = Parallel(n_jobs=-2)(
1374 delayed(get_ccg_sig)(*ccgsig_args[i]) for i in tqdm(range(len(ccgsig_args)),
1375 desc=f'Looking for significant CCGs over {num_cores} cores'))
1377 sigustack=[]
1378 sigstack=[]

File ~\miniconda3\envs\npyx\lib\site-packages\joblib\parallel.py:2007, in Parallel.call(self, iterable)
2001 # The first item from the output is blank, but it makes the interpreter
2002 # progress until it enters the Try/Except block of the generator and
2003 # reach the first yield statement. This starts the aynchronous
2004 # dispatch of the tasks to the workers.
2005 next(output)
-> 2007 return output if self.return_generator else list(output)

File ~\miniconda3\envs\npyx\lib\site-packages\joblib\parallel.py:1650, in Parallel._get_outputs(self, iterator, pre_dispatch)
1647 yield
1649 with self._backend.retrieval_context():
-> 1650 yield from self._retrieve()
1652 except GeneratorExit:
1653 # The generator has been garbage collected before being fully
1654 # consumed. This aborts the remaining tasks if possible and warn
1655 # the user if necessary.
1656 self._exception = True

File ~\miniconda3\envs\npyx\lib\site-packages\joblib\parallel.py:1754, in Parallel._retrieve(self)
1747 while self._wait_retrieval():
1748
1749 # If the callback thread of a worker has signaled that its task
1750 # triggered an exception, or if the retrieval loop has raised an
1751 # exception (e.g. GeneratorExit), exit the loop and surface the
1752 # worker traceback.
1753 if self._aborting:
-> 1754 self._raise_error_fast()
1755 break
1757 # If the next job is not ready for retrieval yet, we just wait for
1758 # async callbacks to progress.

File ~\miniconda3\envs\npyx\lib\site-packages\joblib\parallel.py:1789, in Parallel._raise_error_fast(self)
1785 # If this error job exists, immediatly raise the error by
1786 # calling get_result. This job might not exists if abort has been
1787 # called directly or if the generator is gc'ed.
1788 if error_job is not None:
-> 1789 error_job.get_result(self.timeout)

File ~\miniconda3\envs\npyx\lib\site-packages\joblib\parallel.py:745, in BatchCompletionCallBack.get_result(self, timeout)
739 backend = self.parallel._backend
741 if backend.supports_retrieve_callback:
742 # We assume that the result has already been retrieved by the
743 # callback thread, and is stored internally. It's just waiting to
744 # be returned.
--> 745 return self._return_or_raise()
747 # For other backends, the main thread needs to run the retrieval step.
748 try:

File ~\miniconda3\envs\npyx\lib\site-packages\joblib\parallel.py:763, in BatchCompletionCallBack._return_or_raise(self)
761 try:
762 if self.status == TASK_ERROR:
--> 763 raise self._result
764 return self._result
765 finally:

AttributeError: module 'scipy.signal' has no attribute 'triang'

Neuropyxels with Open Ephys data

Hi,
This is super cool although I can't use it because I'm not using SpikeGL. Is there any chance to have it working with data acquired using Open Ephys?
Best,
Javier

extract_rawChunk bug for plotting raw traces

Hi everyone,

The following error stack issues when plotting raw traces from the example page / readme:

Traceback (most recent call last):
  File "C:\Miniconda3\envs\neuropyxels\lib\site-packages\IPython\core\interactiveshell.py", line 3552, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-cf7a07345f46>", line 1, in <module>
    runfile('C:/Users/diogo/careylab/code/ddcareylab/ephys/python_scripts/trying_neuropyxels.py', wdir='C:/Users/diogo/careylab/code/ddcareylab/ephys/python_scripts')
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.3.5\plugins\python-ce\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.3.5\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/diogo/careylab/code/ddcareylab/ephys/python_scripts/trying_neuropyxels.py", line 64, in <module>
    med_sub=1, whiten=1, hpfilt=1)
  File "C:\Miniconda3\envs\neuropyxels\lib\site-packages\npyx\plot.py", line 1003, in plot_raw_units
    whiten, med_sub, hpfilt, hpfiltf, nRangeWhiten, nRangeMedSub, ignore_ks_chanfilt)
  File "C:\Miniconda3\envs\neuropyxels\lib\site-packages\npyx\inout.py", line 628, in extract_rawChunk
    fname = get_binary_file_path(meta, filt_key)
  File "C:\Miniconda3\envs\neuropyxels\lib\site-packages\npyx\inout.py", line 485, in get_binary_file_path
    relative_path = meta[filt_key]['binary_relative_path']
KeyError: 'ap'

and the culprit seems to be a non-flexible input of extract_rawChunk (4th arg)

which was coded like this:

rc = extract_rawChunk(dp, times, channels, 'ap', saveData,
                          whiten, med_sub, hpfilt, hpfiltf, nRangeWhiten, nRangeMedSub, ignore_ks_chanfilt)

I changed locally to:

rc = extract_rawChunk(dp, times, channels, 'highpass', saveData,
                          whiten, med_sub, hpfilt, hpfiltf, nRangeWhiten, nRangeMedSub, ignore_ks_chanfilt)

and the traces plotted after that.
Still, is there a better way of doing this, and could you please add a fix?

Thanks,
Diogo

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.