Comments (14)
@weiglszonja I just went ahead and transferred it
from pynwb.
@weiglszonja could you please upload the file to https://drive.google.com/drive/u/0/folders/197eVWGgqSItz5DethQdEjlyw_-iCyRSY if you can, so that I can troubleshoot? Thanks.
Strange that
from pynwb import validate
nwbfile_path = "C6-J588-Disc5.nwb"
print(validate(paths=[nwbfile_path], verbose=True))
and
from pynwb import validate
from pynwb import NWBHDF5IO
nwbfile_path = "C6-J588-Disc5.nwb"
with NWBHDF5IO(nwbfile_path, load_namespaces=True) as io:
print(validate(io=io, verbose=True))
have different results. I suspect that one is not loading the namespaces correctly.
from pynwb.
We just discussed with @bendichter that this is more likely a pynwb validation error, I'll open an issue there.
from pynwb.
The ndx-miniscope extension defines a custom Device here: https://github.com/catalystneuro/ndx-miniscope/blob/main/spec/ndx-miniscope.extensions.yaml#L2-L4
io.write
works fine, even io.read
; but it seems pynwb.validate
does not like the subtype of Device linked to the ImagingPlane?
@weiglszonja Can you share the full traceback of pynwb.validate
and figure a way to share the file easily with @rly or @oruebel?
from pynwb.
When you do io.read
what does nwbfile.get_imaging_plane('ImagingPlane').device
return? I'm just wondering, because the validator seems to indicate None
.
from pynwb.
@oruebel @CodyCBakerPhD Thank you for helping figuring out this issue,
nwbfile.get_imaging_plane("ImagingPlane").device
returns
Miniscope abc.Miniscope at 0x5828172864
Fields:
compression: FFV1
deviceType: Miniscope_V3
frameRate: 15FPS
framesPerFile: 1000
gain: High
led0: 47
I just tried this on the file:
from pynwb import validate
nwbfile_path = "C6-J588-Disc5.nwb"
print(validate(paths=[nwbfile_path], verbose=True))
returns
Validating /Users/weian/catalystneuro/demo_notebooks/C6-J588-Disc5.nwb against cached namespace information using namespace 'ndx-miniscope'.
([], 0)
So the file passed the validation then?
But running nwbinspector
again:
import nwbinspector
nwbfile_path = "C6-J588-Disc5.nwb"
print(list(nwbinspector.inspect_nwbfile(nwbfile_path))[0])
returns
InspectorMessage(
message='missing data type Device (device)',
importance=<Importance.PYNWB_VALIDATION: 3>,
severity=<Severity.LOW: 1>,
check_function_name='ImagingPlane',
object_type=None,
object_name=None,
location='general/optophysiology/ImagingPlane',
file_path='/Users/weian/catalystneuro/demo_notebooks/C6-J588-Disc5.nwb'
)
from pynwb.
from pynwb import validate
nwbfile_path = "C6-J588-Disc5.nwb"
print(validate(paths=[nwbfile_path], verbose=True))
The call pattern in the Inspector is actually slightly different; https://github.com/NeurodataWithoutBorders/nwbinspector/blob/dev/src/nwbinspector/nwbinspector.py#L555
Can you try again using the classical way of doing the validation?
with NWBHDF5IO(...) as io:
pynwb.validate(io=io)
from pynwb.
you're right @CodyCBakerPhD !
from pynwb import validate
from pynwb import NWBHDF5IO
nwbfile_path = "C6-J588-Disc5.nwb"
with NWBHDF5IO(nwbfile_path, load_namespaces=True) as io:
print(validate(io=io, verbose=True))
returns
[ImagingPlane (general/optophysiology/ImagingPlane): missing data type Device (device)]
from pynwb.
Thank you @rly , I uploaded a small file.
from pynwb.
Code snippet 1) above results in no validation errors - the correct behavior - because validate
by default will validate against all cached namespaces and ignore dependent namespaces. In this case, the only namespace that will be validated against is ndx-miniscope.
Code snippet 2) above results in a validation error because it only validates against a namespace
arg if provided or the core namespace otherwise. In this case, it will validate against the core namespace, and that validation appears not to understand that data type Miniscope
extends data type Device
. Adding a namespace
arg -- print(validate(io=io, verbose=True, namespace="ndx-miniscope"))
-- results in no validation errors like in case 1.
This raises a few questions.
-
What should the default behavior be for
validate
when theio
arg is passed? I think it should be changed to the same default behavior as when thepaths
arg is passed where the IO object is validated against all cached namespaces except for dependent ones. It looks like when we last updatedpynwb.validate
, we decided to keep the existing behavior ofvalidate
withio
passed to maintain backward compatibility with previous versions of pynwb. #1511 (comment) . The new split behavior is confusing and I think it would be OK to change this behavior. -
Why does validation on the core namespace not understand the type of the
Miniscope
device? "ndx-miniscope" was loaded. It looks like inValidatorMap
, only the types in the given namespace are processed. However, this will result in problems when there are two extensions A and B that independently extend the core namespace. Validation of the file on namespace A will not know about data types in B and vice versa. -
Is there any use case for not simply validating on all the loaded namespaces together? Perhaps both
ValidatorMap
andpynwb.validate
should be updated such that all the loaded namespaces are validated together and you cannot specify a single namespace out of all the ones loaded to validate against. @oruebel @CodyCBakerPhD What do you think? This would be a major breaking change, but I think that is OK.
from pynwb.
@rly do you have a solution how to fix this? This validation error is triggered for any file that is using ndx-miniscope
, so this very much needs a fix. Is there something I can do in ndx-miniscope
?
from pynwb.
@rly @oruebel Any recommendations for this issue?
from pynwb.
@weiglszonja @CodyCBakerPhD I created NeurodataWithoutBorders/nwbinspector#425 which should address the issue quickly.
I think to address the issue in PyNWB without making the change above, we would have to make a breaking change to the behavior of pynwb.validate(io=...)
. (One could argue that the current behavior is a bug, but this behavior has been around for so long and was kept the last time we made changes to the validator, that I am concerned that some downstream code relies on it working this way).
I am in full support of making this change and created #1807 for it. To do this right, I would also want to make a change in the core validation code in HDMF to validate against all extension namespaces. That change may take a little time to implement, so, in the meantime, I recommend changing the code in nwbinspector to use validate(paths=...)
instead of validate(io=...)
as is done in my PR.
Would that resolve the issue at hand?
from pynwb.
@rly Sounds great! Thanks for summarizing so clearly
from pynwb.
Related Issues (20)
- [Bug]: date of birth date time round trip HOT 2
- [Bug]: nwbfile.add_trial_column does not allow extra kwargs HOT 3
- [Bug]: Stimulus API group not finding all objects HOT 3
- [Bug]: Exception reading linked NWBFile with `TimeSeriesReference` in `NWBFile.trials` HOT 10
- [Bug]: Can't use classes created in the same extension as references (link or include) HOT 3
- editing nwb file -- is the recommended way to use h5py? HOT 1
- [Feature]: Make `get_data_in_units` not load entire array into memory HOT 1
- What should be default name of `pynwb.misc.Units`? HOT 2
- [Bug]: IndexSeries: Linked Images has no parent. HOT 3
- Question: why is `grid_spacing` not a field in `ImagingPlane`? HOT 2
- [Bug]: Pre-release workflows failing due to numpy release 2.0.0
- [Feature]: Streaming DANDI:000541 takes a long time
- [Bug]: Check sphinx links workflow failing because of new sphinx warning
- [Bug]: Numpy 2.0
- [Feature]: ros3 streaming on osxarm64
- [Bug]: `epoch_tags` not written to disk HOT 2
- [Feature]: Validation should check for uniqueness of object IDs in file HOT 1
- [Bug]: Cannot add electrodes to existing nwbfile in Zarr format HOT 1
- [Documentation]: Refactor Configuration Tutorial HOT 2
- [Bug]: NWB file created with pynwb cannot be read by OpenEphys HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pynwb.