Code Monkey home page Code Monkey logo

morphio's Introduction

doc/source/logo/BBP-MorphIO.jpg

license documentation status

MorphIO

Documentation

MorphIO documentation is built and hosted on readthedocs.

Introduction

MorphIO is a library for reading and writing neuron morphology files. It supports the following formats:

  • SWC
  • ASC (aka. neurolucida)
  • H5 v1
  • H5 v2 is not supported anymore, see H5v2

It provides 3 C++ classes that are the starting point of every morphology analysis:

  • Soma: contains the information related to the soma.
  • Section: a section is the succession of points between two bifurcations. To the bare minimum the Section object will contain the section type, the position and diameter of each point.
  • Morphology: the morphology object contains general information about the loaded cell but also provides accessors to the different sections.

One important concept is that MorphIO is split into a read-only part and a read/write one.

H5v2

Starting at version 2.6.0, the file format h5v2 is no longer supported. If you have morphologies in this format, you can convert them to h5v1 with:

pip install "morphio<2.6" "morph-tool==2.3.0"

and then:

# single file, OUTPUT must end with `.h5`
morph-tool convert file INPUTFILE OUTPUT
# bulk conversion
morph-tool convert folder -ext h5 INPUTDIR OUTPUTDIR

Contributing

If you want to improve the project or you see any issue, every contribution is welcome. Please check the contribution guidelines for more information.

Acknowledgements

The development of this software was supported by funding to the Blue Brain Project, a research center of the École polytechnique fédérale de Lausanne (EPFL), from the Swiss government’s ETH Board of the Swiss Federal Institutes of Technology.

This research was supported by the EBRAINS research infrastructure, funded from the European Union’s Horizon 2020 Framework Programme for Research and Innovation under the Specific Grant Agreement No. 945539 (Human Brain Project SGA3).

License

MorphIO is licensed under the terms of the Apache License 2.0. See LICENSE.txt for further details.

Copyright (c) 2013-2024 Blue Brain Project/EPFL

morphio's People

Contributors

1uc avatar adevress avatar adrien-berchet avatar alex4200 avatar alexsavulescu avatar alkino avatar arsenius7 avatar asanin-epfl avatar bbpgithubaudit avatar chevtche avatar eile avatar eleftherioszisis avatar favreau avatar ferdonline avatar haleepfl avatar helveg avatar kaabimg avatar matz-e avatar mgeplf avatar musicinmybrain avatar nadirrogue avatar penguinpee avatar ptoharia avatar sanjayankur31 avatar sergiorg-hpc avatar snigdhad avatar st4rl3ss avatar tomdele avatar tribal-tec avatar wizmer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

morphio's Issues

Support Neurolucida multiple soma stack

As in:

  (Color DarkMagenta)
  (CellBody)
  (   -3.19    -9.26     0.08     0.17)  ;  1, 1
  (   -3.53    -9.60     0.08     2.35)  ;  1, 2
  (   -1.01   -10.61     0.08     2.35)  ;  1, 3
  (    0.50   -11.11     0.08     2.35)  ;  1, 4
 )  ;  End of contour

("CellBody"
  (Color Orange)
  (CellBody)
  (    5.88     0.84    54.62     2.35)  ;  2, 1
  (    6.05     2.53    54.62     2.35)  ;  2, 2
  (    6.39     4.38    54.62     2.35)  ;  2, 3
  (    5.55     5.05    54.62     2.35)  ;  2, 4
  (    2.86     5.72    54.62     2.35)  ;  2, 5
)  ;  End of contour

("CellBody"
  (Color RGB (255, 4, 255))
  (CellBody)
  (    1.85     0.67   -90.22     2.35)  ;  3, 1
  (    0.84     1.52   -90.22     2.35)  ;  3, 2
  (   -4.54     2.36   -90.22     2.35)  ;  3, 3
  (   -6.55     0.17   -90.29     2.35)  ;  3, 4
) ; End of contour

ASC Morphology loading issue when using gcc

Issue

Morphology loading of a test .asc file leads to: Error in python': free(): invalid pointer:`.
Problem doesn't seem to present itself if intel compiler is used instead.

Full backtrace is included at the end.

Steps to reproduce

  1. source spack configuration script;
  2. load python-dev/03 gcc cmake boost py-morphio neurodamus-neocortex
  3. create and activate virtual environment and installing dependencies
  4. run the code

The code I was trying to run can be found here: https://bbpcode.epfl.ch/code/#/c/49899/
Error is signaled on line 36 of circuit_indexer.py when running python setup.py test

###Full Backtrace

tests/test_index_mvd.py::test_serial_exec *** Error in `python': free(): invalid pointer: 0x00007fffcb84e1a0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81609)[0x7fffec2f8609]
/gpfs/bbp.cscs.ch/home/bellotta/proj/SpatialIndex/spatial_index/_spatial_index.cpython-37m-x86_64-linux-gnu.so(_ZNSt6locale5_ImplD1Ev+0x7d)[0x7fffcb551ccd]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/compilers/2020-02-01/linux-rhel7-x86_64/gcc-4.8.5/gcc-8.3.0-r24zbv3waz/lib64/libstdc++.so.6(_ZNSt6localeD1Ev+0x34)[0x7fffcaf276b4]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/applications/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-morphio-2.3.4-hk245asvbb/lib/python3.7/site-packages/morphio.cpython-37m-x86_64-linux-gnu.so(_ZN7morphio10MorphologyC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEj+0xaeb)[0x7fffcab9281b]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/applications/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-morphio-2.3.4-hk245asvbb/lib/python3.7/site-packages/morphio.cpython-37m-x86_64-linux-gnu.so(+0x7fc24)[0x7fffcab68c24]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/applications/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-morphio-2.3.4-hk245asvbb/lib/python3.7/site-packages/morphio.cpython-37m-x86_64-linux-gnu.so(+0x9afcb)[0x7fffcab83fcb]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyMethodDef_RawFastCallDict+0x132)[0x7fffed43f7a2]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyCFunction_FastCallDict+0x25)[0x7fffed43f9f5]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyObject_Call_Prepend+0xcd)[0x7fffed43fd3d]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(PyObject_Call+0x75)[0x7fffed440f45]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(+0xee076)[0x7fffed499076]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(+0xea6b2)[0x7fffed4956b2]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyObject_FastCallKeywords+0xd3)[0x7fffed43f3d3]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x5bdd)[0x7fffed419d5d]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(+0x67feb)[0x7fffed412feb]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x7995)[0x7fffed41bb15]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(+0x67feb)[0x7fffed412feb]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x7995)[0x7fffed41bb15]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(+0x67feb)[0x7fffed412feb]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x7995)[0x7fffed41bb15]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyEval_EvalCodeWithName+0x97e)[0x7fffed51447e]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyFunction_FastCallKeywords+0x93)[0x7fffed43ecf3]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x7995)[0x7fffed41bb15]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(+0x67feb)[0x7fffed412feb]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x79aa)[0x7fffed41bb2a]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(+0x67feb)[0x7fffed412feb]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyFunction_FastCallDict+0x2d2)[0x7fffed43ec52]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x2dbf)[0x7fffed416f3f]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyEval_EvalCodeWithName+0x97e)[0x7fffed51447e]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0(_PyFunction_FastCallDict+0xae)[0x7fffed43ea2e]
/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/libpython3.7m.so.1.0Fatal Python error: Aborted

Current thread 0x00007fffedad7740 (most recent call first):
  File "/gpfs/bbp.cscs.ch/home/bellotta/proj/SpatialIndex/spatial_index/circuit_indexer.py", line 36 in _load
  File "/gpfs/bbp.cscs.ch/home/bellotta/proj/SpatialIndex/spatial_index/circuit_indexer.py", line 48 in get
  File "/gpfs/bbp.cscs.ch/home/bellotta/proj/SpatialIndex/spatial_index/circuit_indexer.py", line 58 in process_cell
  File "/gpfs/bbp.cscs.ch/home/bellotta/proj/SpatialIndex/spatial_index/circuit_indexer.py", line 80 in process_range
  File "/gpfs/bbp.cscs.ch/home/bellotta/proj/SpatialIndex/spatial_index/circuit_indexer.py", line 106 in main_serial
  File "/gpfs/bbp.cscs.ch/home/bellotta/proj/SpatialIndex/tests/test_index_mvd.py", line 17 in test_serial_exec
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pytest-5.2.1-wwksimpf7d/lib/python3.7/site-packages/_pytest/python.py", line 170 in pytest_pyfunc_call
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pytest-5.2.1-wwksimpf7d/lib/python3.7/site-packages/_pytest/python.py", line 1423 in runtest
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pytest-5.2.1-wwksimpf7d/lib/python3.7/site-packages/_pytest/runner.py", line 125 in pytest_runtest_call
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pytest-5.2.1-wwksimpf7d/lib/python3.7/site-packages/_pytest/runner.py", line 201 in <lambda>
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pytest-5.2.1-wwksimpf7d/lib/python3.7/site-packages/_pytest/runner.py", line 229 in from_call
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pytest-5.2.1-wwksimpf7d/lib/python3.7/site-packages/_pytest/runner.py", line 201 in call_runtest_hook
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pytest-5.2.1-wwksimpf7d/lib/python3.7/site-packages/_pytest/runner.py", line 176 in call_and_report
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pytest-5.2.1-wwksimpf7d/lib/python3.7/site-packages/_pytest/runner.py", line 95 in runtestprotocol
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pytest-5.2.1-wwksimpf7d/lib/python3.7/site-packages/_pytest/runner.py", line 80 in pytest_runtest_protocol
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pytest-5.2.1-wwksimpf7d/lib/python3.7/site-packages/_pytest/main.py", line 256 in pytest_runtestloop
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pytest-5.2.1-wwksimpf7d/lib/python3.7/site-packages/_pytest/main.py", line 235 in _main
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pytest-5.2.1-wwksimpf7d/lib/python3.7/site-packages/_pytest/main.py", line 191 in wrap_session
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pytest-5.2.1-wwksimpf7d/lib/python3.7/site-packages/_pytest/main.py", line 228 in pytest_cmdline_main
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pluggy-0.13.0-wo7mj37ztp/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-pytest-5.2.1-wwksimpf7d/lib/python3.7/site-packages/_pytest/config/__init__.py", line 90 in main
  File "/gpfs/bbp.cscs.ch/home/bellotta/proj/SpatialIndex/.eggs/pytest_runner-5.2-py3.7.egg/ptr.py", line 220 in run_tests
  File "/gpfs/bbp.cscs.ch/home/bellotta/proj/SpatialIndex/.eggs/pytest_runner-5.2-py3.7.egg/ptr.py", line 209 in run
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/python3.7/distutils/dist.py", line 985 in run_command
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/python3.7/distutils/dist.py", line 966 in run_commands
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/external-libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/python-3.7.4-tfxecymrkn/lib/python3.7/distutils/core.py", line 148 in setup
  File "/gpfs/bbp.cscs.ch/ssd/apps/hpc/jenkins/deploy/libraries/2020-02-01/linux-rhel7-x86_64/gcc-8.3.0/py-setuptools-41.4.0-ooqzubldp5/lib/python3.7/site-packages/setuptools/__init__.py", line 145 in setup
  File "setup.py", line 146 in setup_package
  File "setup.py", line 151 in <module>
[1]    50986 abort      python setup.py test

Change of diameter only effective when all diameters are updated

If one does section.diameter[0] = 1, it will have no effect, but one has to pass the whole array of diameters.
Could it be possible to change the diameter of a single point, or at a least display a warning to the user doing that without knowing? (I encountered this twice, and second time I forgot about it...)

Cleanup: organize headers

For example, in src/plugin/morphologySWC.cpp, the header order is reversed: system headers come before "morphologySWC.h", for instance.

We should decide on 'correct' way, and apply that across the whole project.

NEURON ordering option in write mutable morphology

MorphIO allows to open a morphology using a specific ordering.

However, it is not possible to write a synthesized cell using such an ordering.

For the time being the workaround to achieve this is to write the cell, open it in readonly mode with NEURON ordering, convert it to the mutable data structure and write it again.

Is it possible to allow the user specify the ordering for writing the morphology to file?

Why providing default deleter when creating shared_ptr?

While reading some code in MorphIO, I am wondering why some shared_ptr are constructed with a deleter object calling ::delete which is already the default behavior:

void friendDtorForSharedPtrMito(MitoSection* section)
{
delete section;
}
std::shared_ptr<MitoSection> Mitochondria::appendRootSection(
const morphio::MitoSection& section_, bool recursive)
{
std::shared_ptr<MitoSection> ptr(new MitoSection(this, _counter, section_),
friendDtorForSharedPtrMito);

Is there something that prevent using std::make_shared?

const auto ptr = std::make_shared<MitoSection>(this, _counter, section_);

Access to connectivity information without generating the sections (immutable)

My understanding for the immutable Morphology is that the section connectivity is stored in a table.

Currently, one can access the morphology points and properties, such as diameters, but there is not way to access the section connectivity from the morphology level.

The section connectivity could be returned in the form of an edge list e.g.

from morphio import Morphology
morphology = Morphology(path)
section_connectivity = morphology.section_connectivity

which would return an array like the following:

[[0, 1], [1, 2], [1, 3]]

Expanding on this, given that the section has an associated id, then we could retrieve the specific section by id, while having the section node adjacency.

Returning also the point connectivity can be useful for graph analysis. For example:

from morphio import Morphology
morphology = Morphology(path)
points = morphology.points
point_connectivity = morphology.point_connectivity

This would allow to directly convert the morphology point connectivity into an adjacency matrix of the vertices / points.

two possible improvements for vasculature datasets

It would be nice to have the following two features in MorphIO, related to working with vasculature datasets and blood flow simulations

  1. File format

The original data for vasculatures is in .vtk format, with two tables:

  • list of points
  • list of segments with radii associated with them

This format seems not compatible with the hdf5 files red by MorphIO, as they have radii values on points. The conversion from vtk to hdf5, therefore, modifies the original data. In addition, it is not invertible, i.e. one cannot go back fro hdf5 to the original vtk.

Reading the hdf5 vasculature doc a bit more carefully, (https://bbpteam.epfl.ch/documentation/projects/Morphology%20Documentation/latest/h5vasculature.html) I noticed the optional group/property, which can assign any values on points, segments or sections.

So here is a possible option to solve this. We can use this additional section of the hdf5 format to save the original radii on segments so that we are faithful to the original datasets.

In addition, for blood flow simulations, some results will be on segments (blood flow) or points (blood pressure), and possibly time/frequency-dependent. This implies that multiple columns should be available for points and segment optional array.

  1. Connectivity format

If MorphIO could load the hdf5 format into a connectivity graph based on points, rather than sections, possibly of the same format as in vtk files, i.e. list of points (same as in hdf5), and list of segments (two points ids per segment), with corresponding radii (taken from the original data, see point above), or any other values.

check_clang_format.sh does not run locally

./ci/check_clang_format.sh                                                                                                             
./ci/check_clang_format.sh: line 13: venv-clang-format/bin/pip: No such file or directory

as python3 -mvenv "$VENV" does not seem to provide a pip

Environment:

Python 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0] on linux
Ubuntu 18.04.2 LTS
fish, version 2.7.1

Use C++14

Is this normal that the tests use C++14 but the library C++11 ?

src/CMakeLists.txt:  CXX_STANDARD 11
src/CMakeLists.txt:  CXX_STANDARD_REQUIRED YES
tests/CMakeLists.txt:  CXX_STANDARD 14
tests/CMakeLists.txt:  CXX_STANDARD_REQUIRED YES

According to the TravisCI configuration, the oldest supported compiler seems to be GCC 5.4.0, in Ubuntu xenial. This compiler version supports C++14, see compatibility table

So I don't see a reason not to switch to 14 and embrace make_unique, polymorphic lambda expressions, variable template, and much more.

Besides, full support to 17 on BB5 is on its way \o/

Inconsistency in missing parent exception between readonly and mut

import morphio
m = morphio.Morphology(filename).root_sections[0].parent

raises:

MissingParentError: Cannot call Section::parent() on a root node (section id=0).

while:

import morphio
m = morphio.mut.Morphology(filename).root_sections[0].parent

raises:

IndexError: map::at

It is important for readonly and mut to exhibit similar behaviour so that they can be used by algorithms interchangeably.

Add CMAKE flag for disabling unifurcation sanitization

Following a discussion with @mgeplf the other day, I think it may be worth it to temporarily add a CMAKE flag to disable the unifurcation sanitization to be able to easily work with circuits that already have them (and until the edges and morphologies in those circuits get updated).

What do you think @mgeplf ?

Use `size_t` instead of `uint32_t`

I was wondering if it would be preferable to use the more general and less constraining size_t where unsinged int is explicitly used as a type (mainly ids).

I don't see any reason in constraining the implementation to 32 bit ints. Thoughts?

Inconsistency between H5 ans SWC writer

When using morphio to write a neuron, I noticed that the SWC and H5 files I wrote were behaving differently in a subsequent analysis:

neurom.get('trunk_angles', n1)                                                                                                                                  
array([1.22883997, 2.40792416, 1.24811592, 0.20883851, 0.04060654, 1.14886118])

neurom.get('trunk_angles', n2)
array([nan, 0.39250956, 0.24012936, 0.12505715, 0.87182467,  nan])

I suspect this is due to an inconsistency at the first points of the trees.

I attach the example files that were expected to be the same Example.zip

Set single value in mutable section

Allow to mutate the values of the section data without copying and rewriting the entire datasets every time we want to change a single value.

The most basic example would be like this (location in the vector followed by value):

section.set_diameter(5, 0.1)
section.set_point(0, [0.1, 0.2, 0.3])
section.set_perimeter(2, 2.0)

Setting a section property array (like point or diameter) changes its address

The latest morphio version has section properties to not return a copy in order to allow for single property setting.

However, this has the drawback that an initial local variable with the values is deallocating its memory before the end of the function scope if new values are set.

Any function that uses morphio and expects the getter to return values that persist until the end of the scope or change wrt to the new values, will lead to numerical errors.

I am afraid that this feature could possibly silently break tools that rely on morphio. If it weren't for my tests failing I wouldn't have realized this is taking place.

Example:

m = morphio.mut.Morphology(...)
s = m.sections[0]
a = s.points
a_copy = np.copy(a)
s.points = np.random.random((len(s.points, 3))
b = s.points

assert np.allclose(a, a_copy)
assert np.allclose(a, b)

I would expect that the second assertion should not fail.

Re-using existing build directories leads to hard to debug failures

We currently default to a "standard" build directory for the Python C++ extension:

if not os.path.exists(self.build_temp):

When building repeatedly, and changing the environment while doing so, re-using the default build directory can lead to outdated CMake caches and wrong builds. We should make sure that we either build in a temporary directory or clean the build directory before building ourselves. I'm in favor of the former. @mgeplf, opinions?

No warning for one point root_section

While there can be a section with just one point and the duplicate one, this is not true for root sections that do not have duplicate points. In MorphIO you can write a file with a single point root section, without emitting a warning.

from morphio import Morphology
from morphio import PointLevel, SectionType 

m = Morphology()

points = [[1., 1., 1.]
diameters = [2.]

m.append_root_section(PointLevel(points, diameters), SectionType(2)) 
m.write('test_neuron.h5')

This doesn't produce any warnings when it is written.

HDF5 "badbadneuron.h5" {
GROUP "/" {
   ATTRIBUTE "comment" {
      DATATYPE  H5T_STRING {
         STRSIZE H5T_VARIABLE;
         STRPAD H5T_STR_NULLTERM;
         CSET H5T_CSET_UTF8;
         CTYPE H5T_C_S1;
      }
      DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
      DATA {
      (0): "Created by MorphIO v2.0.7"
      }
   }
   GROUP "metadata" {
      ATTRIBUTE "cell_family" {
         DATATYPE  H5T_STD_U32LE
         DATASPACE  SIMPLE { ( 1 ) / ( 1 ) }
         DATA {
         (0): 0
         }
      }
      ATTRIBUTE "version" {
         DATATYPE  H5T_STD_U32LE
         DATASPACE  SIMPLE { ( 2 ) / ( 2 ) }
         DATA {
         (0): 1, 1
         }
      }
   }
   DATASET "points" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 1, 4 ) / ( 1, 4 ) }
      DATA {
      (0,0): 1, 1, 1, 2
      }
   }
   DATASET "structure" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SIMPLE { ( 2, 3 ) / ( 2, 3 ) }
      DATA {
      (0,0): 0, 1, -1,
      (1,0): 0, 2, 0
      }
   }
}
}

Assigning values from non-contiguous array

It seems that section points assignment is confused with non-C style arrays:

>> pp = np.array([[1, 2], [3, 4], [5, 6]]).T
>> pp
[[1, 3, 5], [2, 4, 6]]
>> pp.strides
(8, 16)
>> s.points = pp
>> s.points
[[1, 2, 3], [4, 5, 6]]

Tested with 2.0.0 wheel as I could not install dev version of BB5 (see also #26).

Absorbing of single child sections

Hello! I want to propose an update to API. Currently when MorphIO merges single child section into parent, user receives only a print warning message of WARNING_ONLY_CHILD. Shouldn't user receive en exception with the section being deleted and its parent section that absorbs it? Or provide a mapping between deleted sections and their new sections?

Vasculature section connectivity

Similarly to the neuronal morphologies, the section connectivity is also useful for vasculature datasets.

from morphio.vasculature import Vasculature
v = Vasculature(filepath)
v.connectivity

Resampling function

I would like to request a new feature to be implemented and executed during the loading of vascular morphology. This function resamples each section in the morphology to remove unwanted or unnecessary samples. The implementation in C++ would be much faster than python and would save substantial amount of time whe loading large scale morphologies in VessMorphoVis.

You can see some examples in the attached images.

Screenshot from 2020-06-04 18-34-58

Protected constructors vs performances

It would be more efficient to use std::vector::emplace_back to fill variable sections_ instead of creating an additional copy of every object.

const MitoSection Mitochondria::section(const uint32_t& id) const
{
return MitoSection(id, _properties);
}
const std::vector<MitoSection> Mitochondria::sections() const
{
std::vector<MitoSection> sections_;
for (unsigned int i = 0;
i < _properties->get<morphio::Property::MitoSection>().size(); ++i) {
sections_.push_back(section(i));
}
return sections_;
}

Something like:

   std::vector<MitoSection> sections;
    for (unsigned int i = 0;
         i < _properties->get<morphio::Property::MitoSection>().size(); ++i) {
        sections.emplace_back(i, _properties);
    }

But std::vector::emplac_back is not able to instantiate MitoSection objects because the constructor is protected.
There are workarounds for such situation (see stackoverflow article) but it complicates things quite a lot.

So I am wondering if it is ok to mark the constructor public and mention in a comment that this class is not supposed to be instantiated directly.

Differences between mutable and immutable API

The natural expectation for mut.Morphology API is to be a superset of Morphology.
However, the are some things that are different:

  1. No .points property (is it due to different internal representation? but nevertheless)
  2. .sections seem to work differently; for instance for simple.asc from test data, it's a list of 6 items for immutable one; and 5 items for mutable one.

Tested with 2.0.0 wheel as I could not install dev version of BB5 (see also #26).

Writing morphology with perimeters as ascii does not throw an error

ascii format does not support perimeters (only h5). In that case shouldn't morphio raise an error if someone (like me) mistakenly wrote the file in a different format?

Example:

import morphio
m = morphio.Morphology('test.asc').as_mutable()

for section in m.iter():
    section.perimeters = section.diameters

m.write('test_perimeters.asc')

It silently drops perimeters and writes the file.

Starting point index of each section

Hello,

Currently I am trying to inject the morphology objects from MorphIO API into our package SpatialIndex. The SpatialIndex API has a method to add branches with an array of points (start/end points of segments), an array of radii, and an array of the first point index of each branch. For the first two arrays, I can use the property accessors of the MorphIO API (like Morphology.points, Morphology.diameters).
My question is that is it possible to get the index of the starting points of each section directly from the API? I am trying to avoid loops in Python for performance issue, so it would be better to have such array directly constructed by the API.

Suggestions and discussions are appreciated.

Cheers,
Weina

Add support for double-precision floating-point

We would need to have the possibility of using Morphio with doubles, as well as its python bindings. For example, NRN is using doubles for points and their computation.

After discussing with @wizmer and @ferdonline, we could consider one of the following:

  • CMAKE option to enable double support
  • just use double as default

TBD

Ensure that duplicate point check includes diameters

It would be useful to indicate if the diameters between the last point of a section and the first points of its children are consistent. This will ensure the correctness of duplicate points as far as diameters are concerned.

Writing an h5 results in seg-fault if annotations are present on morphology

Example (swc works but annotations are not written to file, h5 throws seg-fault error)

import morphio
m = morphio.mut.Morphology()
m.annotations.append('{"Apical point": (0, 1, 0)}')

m.write('Test annot.swc')
Warning: writing file without a soma

m.write('Test annot.h5')
Warning: writing file without a soma
Segmentation fault

Symbol visibility issues (Clang)

Compiling for mac I see a lot of these warnings

ld: warning: direct access in function 'bind_immutable_module(pybind11::module&)' from file 'CMakeFiles/_morphio.dir/bind_immutable.cpp.o' to global weak symbol 'morphio::SectionBasemorphio::Section::id() const' from file '../../src/libmorphio_static.a(morphology.cpp.o)' means the weak symbol cannot be overridden at runtime. This was likely caused by different translation units being compiled with different visibility settings.

When Werror is active it fails compilation.

What should we do with "CellBody" elements ?

In ASC format, a soma is usually defined my the starting token (CellBody).
However in some cases, it happens that this token is missing but the "CellBody" token is there instead.

Here is an example extracted from a real file:

("CellBody"
  (Color Green)
  (Closed)
  (GUID "9C6D866499E44219B0F15069BE9FD825")
  (MBFObjectType 5)
  (FillDensity 0)
  (Resolution 0.073746)
  (   35.70    -4.72     8.60     0.07)  ;  4, 1
  (   38.25    -3.55     8.60     0.07)  ;  4, 2
  (   40.20    -2.73     8.60     0.07)  ;  4, 3
  (   42.23    -1.47     8.60     0.07)  ;  4, 4
... the file continues

If I'm correctly interpreting @jdcourcol's sayings, this quoted CellBody is simply a comment written by the user. It is not the official token that should be present.

The question is: Should this kind of soma be valid or not ? I know we had issues when we tried to allow such kind of soma in the past. That's why the following line is commented in the code https://github.com/wizmer/MorphIO/blob/master/src/readers/lex.cpp#L158
I don't remember why though.

Put `sanitize` method to Python API

Hi!

  • Is it possible to put void Morphology::sanitize of mut to the Python accessible API? Currently I can't call it directly from Python API. I have to save the morphology in order to sanitize it.
  • Is it possible to perform sanitize inplace?

Lifetime issues with mut.Morphology constructing sections with raw pointers to itself

Constructs like this:
Section(this, _counter, section_) (where this is mut::Morphology` - from)

This means that a raw pointer is stored here that has a different lifetime than Morphology.

This causes a use after free if the Morphology is dtor'd before the sections that have been created are removed, for example:

from morphio.mut import Morphology
m = Morphology('simple.swc')
s = m.root_sections[0]
del m # ~mut.Morphology() called
print(s.children) # use after free

Note, this is different than #29

Single section vasculature fails to load due to empty connectivity

I tried to load an one section vasculature dataset, but MorphIO doesn't load it due to the empty connectivity dataset.
test_single_section_sg.zip

from morphio.vasculature import Vasculature

v = Vasculature('test_single_section.h5')

Traceback:

---------------------------------------------------------------------------
RawDataError                              Traceback (most recent call last)
<ipython-input-3-43b259418727> in <module>
----> 1 m = morphio.vasculature.Vasculature('test_single_section_sg.h5')

RawDataError: Opening vasculature file 'test_single_section_sg.h5': bad number of dimensions in connectivity dataspace

h5dump:

HDF5 "test_single_section_sg.h5" {
GROUP "/" {
   DATASET "connectivity" {
      DATATYPE  H5T_IEEE_F64LE
      DATASPACE  SIMPLE { ( 0 ) / ( 0 ) }
      DATA {
      }
   }
   DATASET "points" {
      DATATYPE  H5T_IEEE_F32LE
      DATASPACE  SIMPLE { ( 3, 4 ) / ( 3, 4 ) }
      DATA {
      (0,0): 0.685519, 0.68087, 0.39709, 1,
      (1,0): 0.512065, 0.125548, 0.0812268, 1,
      (2,0): 0.832563, 0.221666, 0.261854, 1
      }
   }
   DATASET "structure" {
      DATATYPE  H5T_STD_U64LE
      DATASPACE  SIMPLE { ( 1, 2 ) / ( 1, 2 ) }
      DATA {
      (0,0): 0, 0
      }
   }
}
}

Using `Section` outside of `Morphology` scope

In some usecases, we are interested only in loading and transforming a single neurite from a given Morphology.

A natural flow would be some like

def load_axon(filepath):
    morph = morphio.Morphology(filepath)
    result = morph.root_sections[0]  # for the sake of brevity, let's assume it's axon
    return result

axon = load_axon('blah')
for s in axon.iter():
    ...

It seems though that once morph object goes out of scope, axon is invalidated.
I vaguely remember we saw something like that before.
Is there a mechanism to hold morphology memory representation from Section?

Alternatively, what would work for this particular usecase is loading only axon in the morphology object.
In this case I can operate the whole morphology object (pass it around, transform points, etc.) without caring to extract the neurite of interest.

More radical proposal: sacrifice "view into C++ object memory" paradigm in Python; and make a copy of internal data into Python world on morphology loading. At the cost of data copy, we'll save ourselves from tons of headaches (see also #28, for instance). Getting memory management right in both C++ and Python with the same code can be very tricky.

Tested with 2.0.0 wheel as I could not install dev version of BB5 (see also #26).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.