When analog input signals (channels > 128) are loaded the gain value is a float number by that the integer signal values are multiplied. This multiplication is done in different ways in the old BlackrockIO_v4 and the new BlackRockIO (happening in BaseRawIO, ll. 490 ff.). Both differ from the expected value. The behavior of the old IO can be explained because of the float precision loss whereas I could not reproduce the result of the new IO by trying to multiply manually at different precisions. The differences are at about the seventh significant digit, thus not extreme.

AnalogSignal constructor should throw an error if the two parameters are both given, but contradict. Currently, it just uses one of them and sets the other, which is error-prone.

The user is able to flexibly select data from Neo objects, filtering them by object type, annotation, etc.

Issues to be addressed in the epic:

• current filter function has a difficult syntax, no/bad documentation
• implement multi-filtering -> Filter an object, and then apply another filter
• multiple allowed values: AND’s and OR’s of annotations, combining queries
• exclude values: extract objects that do not have a certain attribute
• support for regular expressions?
• multi annotations –> For AnalogSignal objects, one annotation for each signal in the object. For Events and Epochs, one annotation for each event/epoch.
• Should recognize quantities –> search for something like T_min==1000ms.
• Masking of data within a data object (Note: Robin G found an interesting way to do this)

Time stamp of the first sample of an nsX file segment is not the offset provided by in the file header, but the offset + 1/sampling_frequency (ie. the id of the first sample is 1 and not 0)

Currently, IOs are quite limited in terms of the being able to either load everything (or whatever the IO provides), or simply to lazy load. For that reason, often, one needs to work with two Neo Blocks: a lazy one with the complete Neo Block, and a second Neo Block with only the subset of data of interest.

A better mechanism would be one where each data object could be explicitly loaded or removed from memory (and container objects doing this recursively for all their data children). Since this meachnism would imply that each IO is aware of how to load a specific part of the data, which may require massive IO restructuring, a more straight forward way to do this would be a caching system, where each data object has a "stash" and a "pop" method that temporarily staches the contents to disk and frees memory.

An even more elaborate, but complicated project is one where the stashing and poping is performed on-demand, automatically, in the background.

This epic defines how trials are represented in Neo (and Elephant), allows users to create trials from vanilla Neo objects, and work with them efficiently.

A number of prototypes were already created for the Scientific Data publication and could serve as a starting point.

Current Blackrock IO in Neo seems to have problems with some of our data. In this issue we will

• test the current blackrock IO and reintroduce/enhance the unit tests of version 2
• investigate any limitations in working with the IO regarding feature richness of the current IO due to the uncluttering of the API from V2 to V3.

Work is carried out on branch https://github.com/INM-6/python-neo/tree/feature/new_blackrockio_v4

Branch is based on apibreak branch of upstream.

To reproduce:

a=AnalogSignal([1,2,3]*pq.mV)
b=a.duplicate_with_new_array([4,5,6]*pq.ms)
--> Error: cannot convert mV to ms

AnalogSignals are not carried over.

j=bl.segments[0]
k=bl.filter([{'unit_id': 0},{'electrode_id': 4}],objects="Unit")[0]
print k
l=j.construct_subsegment_by_unit(unit_list=[k])
# This is ok...
print l.spiketrains
# Should not be empty...
print l.analogsignals

Investigate whether pint would be a good substitute for quantities.

This epic provides basic Spiketrain/AnalogSignal operations, such as merge, cut, concat, transpose (y-axis shift), time-shift.

If filtering a neo objects with an empty targdict, then nothing is returned. Desirable would be to return everything.

block.filter(targdict={},objects=["SpikeTrain"])

--> []

Segments don't have a t_start and t_stop.

Ideally, these would be two functions of the Segment that calculate t_start and t_stop as the minimum and maximum times of all children, respectively.

a=neo.SpikeTrain([1,2,3]*pq.ms,t_stop=10*pq.ms)
b=neo.SpikeTrain([1,4,5]*pq.ms,t_stop=10*pq.ms)
print a.merge(b)
None

In this example, t_start is filtered to be 0, but the t_start is actually 0 seconds. Thus, the unit is ignored. Is this behavior desireable (i.e., if not unit is passed when filtering a quantity, assume the unit of the property/annotation?

h=bl.filter({'t_start':0})
print len(h)
print h[0]
# This is 0 seconds, not 0
print h[0].t_start

In the current implementation of the AnalogSignalArray the time is represented in the first axis. This results for example in the extraction of a single AnalogSignal like this:
analogsignal = analogsignalarray[:,0]
instead of the (probably) more intuitive version
analogsignal = analogsignalarray[0]

Another aspect important for this is the actual representation of the data in memory, such that a single analogsignal can be read in one go (Fortran vs. C representation of matrixes in memory). Furthermore input of a single analogsignals of length n in row format should not lead to an interpretation of n analogsignals with one entry each. Therefore input dependent reshaping is needed when creating an analogsignalarray.

A function like this would be useful:

h.construct_subsegment_by_recordingchannel(rc_list=[k])

Investigate if mnespikes [https://pypi.python.org/pypi/mnespikes] can be used as simple interface for spiking data.

Currently, the filter function in neo accepts inputs of the form

targdict=[{cond1,cond2},{cond3,cond4},..]

where cond1 and cond2 are OR, whereas the individual dictionaries {cond1,cond2} and {cond3,cond4} are AND connected.

One drawback is that it is not possible to filter for multiple values. For example, filtering for units 0 and 1 on electrode 7 is not possible via the following statement, which returns only unit 1 (the last filtering condition specified):

bl.filter(targdict=[{"unit_id":0,"unit_id":1},{"electrode_id":7}], objects=neo.core.SpikeTrain)

Output:

[SpikeTrain
 name: 'Segment 0, Channel 7, Unit 1'
 annotations: {'channel_id': 7,
   'electrode_id': 7,
   'mua': False,
   'sua': False,
   'unit_id': 1}]

To achive the desired behavior, you need to use complex list comprehensions:

z=[item for sublist in [bl.filter(targdict=[{"unit_id":z},{"electrode_id":7}],objects=neo.core.SpikeTrain) for z in [0,1]] for item in sublist]

Example:

import neo 
import quantities as pq 
st = neo.SpikeTrain([1, 0, 2, 3]* pq.ms, t_start=0*pq.ms, t_stop=3*pq.ms)
print(st) 
>>> [ 1.  0.  2.  3.] ms

should give an error or correctly sort the spikes.

GDF IO Development

1. is .gdf the correct file format? -> go with it for now, evtl change later
2. number of columns does not tell the content of columns -> default: two column structure (gid, time)
3. time unit of time stamps? -> always ms (3 digits), except if time stamps are given in simulation time steps

TODO:
-Unittests
--4 testcases of input (columns...) (done)
--check if data types (int, float) are read properly (done)
--check specific spike times of single neurons. for one id, load with np.loadtxt and compare spike trains (done)
--test segments: empty list, list of subset of ids + request all neurons... (done)
--wrong user input (done)
--create spike train only for a time interval between t_start and t_stop (done)
--conductance-based neurons for reading out V_m and g_ex
--assign spikes to neurons with a different routine (done)
--upload test data to g-node server (contact Thomas Wachtler).
--run gdfio test based on remote data (see blackrockio for implementation details)

-NESTio
--read more file formats (.dat for membrane potential)
--read_segment() for spikes and analogue signals is currently overwritten

-Bugs
--t_start and t_stop have different units (done)

-Documentation

-NEST core meeting: class hierarchy, which output, which file format, which metadata
--main idea: NESTio shall be able to load all data types that NEST can write

Neo related:
SpikeTrain instance with t_stop = None does not complain directly
ST has inconsistent time handling st([1,2,3]_pq.ms,t_start=10_pq.s,t_stop=0.1) results in t_stop = 0.1*pq.ms
-> job for long/alper/michael session...?

DONES for NESTIO:

• test for multiple value columns (done)
• think about implementation of read_segment() to handle both spikes and analog signals (done)
• documentation of NestIO class (done, Julia) (Johanna)
• solve read_segment conflict between inherited version from gdfio and implemented version in nest_io (done)
• dat test files with time in steps are missing (done)
  other:
• delete other branch feature/gdf_io -> ask Michael (done)
• add test for dat files without gid (done)
• check if tests for all test files exist (done)
• 'dat' and 'gdf' always upper or lowercase in docu? (done)

TODO for NESTIO:

Needs further investigation, but for example, when doing a deep copy of a Block, sometime the annotations of AnalogSignals get lost.

Currently, the documentation of Neo is insufficient.

• Some functions are only documented superficially.
• The Neo manual contains only very rudimentary tutorials. They should better guide users on how to properly work with Neo. This is related to tasks that improve the usability, e.g. #23 #24 #26.

When creating a spike train with different units (or no unit) for t_stop, t_start and the spike times Neo automatically assigns a unit to the non-quantity provided.
Example:
SpikeTrain([1,2,3]_pq.ms, t_stop = 10_pq.s, t_start = 0.1) results in a spike train with t_stop = 0.1ms instead of t_stop = 0.1s. Instead raising an error 'ambiguous quantities provided' would make sense.

Currently, spikes in a spiketrain are not sorted in time. This could be confusing for users. On the other hand, sorting should not just automatically happen, as this may also cause bugs. This issue aims to find a clear, transparent solution (in code and/or documentation) to define if and when Spiketrains are sorted.

Neo 0.5 introduces a number of Neo objects that can contain more than one data item:

• AnalogSignal
• Event
• Epoch
  For example, an Event object may contain multiple recorded event times. However, annotations are limited to the Event object as a whole. Likewise, filtering only works on the complete event objects.

This epic develops the concept to

• provide annotations for individual data items in one such Neo object (called multi-annotations for now)
• and the mechanisms to efficiently work with such multi-annotations

It would be very useful to slice AnalogSignals by time, just like SpikeTrains.

l=bl.list_children_by_class("SpikeTrain")[0]
print l.time_slice(0,50000)
l=bl.list_children_by_class("AnalogSignal")[0]
print l.time_slice(0,5000)

It should also accept, e.g., 'spiketrain' or 'SpiKETRain'.

Current BlackrockIO creates a mess with the ChannelIndex, partly because ChannelIndex is somewhat inconsistent. This issue should fix both, the channelindex object, as well as the way its used in the BlackrockIO:

[ ] Differentiate index/channel_indexes/channel_ids/channel_names
[ ] Create meaningful names of ChannelIndex objects in BlackrockIO
[ ] Make sure only those ChannelIndex objects are created which are used.

based on samuelgarcias comment here: NeuralEnsemble#345 (comment)

I very recently try to use this IO but get into problems with cachedir. I don't remember exaclty.

Since I need only to acces AnalogSignal with original dtype (int16) I rewrote this by myself.

My code is really fast because it use mmap and is able to read chunk very fastly.
The code is here at the end of the file:
https://github.com/samuelgarcia/tridesclous/blob/master/tridesclous/datasource.py

maybe the 2 codes could be merge one day.

There have been a number of changes for blackrockio committed to the blackrockio_fix_nix branch, and in addition there have been a few fixes on that file on the main Neo Repo. This issue is aimed at consolidating the two.