paris-saclay-cds / specio Goto Github PK

View Code? Open in Web Editor NEW

19.0 5.0 13.0 50.09 MB

specio: Python input/output for spectroscopic files

Home Page: http://specio.readthedocs.io/

License: BSD 3-Clause "New" or "Revised" License

Python 98.22% Shell 1.13% Makefile 0.65%

python hyperspectral raman chemistry

specio's People

Contributors

Stargazers

Watchers

Forkers

jorisvandenbossche glemaitre stfaili grseb9s laziszaire hsiyjnd charliekc maxwell0001 forjobs waterwawe jjaconn libertyforeman robna

specio's Issues

Error with the doc

I have an error with the doc when the project is on the drive D: with Windows10.

File "D:\PROJET\VRAC2\specio\doc\sphinxext\github_link.py", line 62, in _linkcode_resolve
    start=os.path.dirname(__import__(package).__file__))
  File "c:\users\daria\appdata\local\programs\python\python37\lib\ntpath.py", line 562, in relpath
    path_drive, start_drive))
ValueError: path is on mount 'D:', start on mount 'c:'

Improve the display of the metadata

Add an argument to read_csv to merge wavelength

Python 3.10 compatibility

The current version is unfortunately no longer compatible with python3.10.X

ImportError: cannot import name 'Iterable' from 'collections'

File /opt/homebrew/lib/python3.10/site-packages/specio/core/__init__.py:6, in <module>
      1 """ This subpackage provides the core functionality of specio
      2 (everything but the plugins).
      3 """
      5 from .util import Spectrum, Dict
----> 6 from .format import Format, FormatManager
      7 from .exceptions import CannotReadSpectraError
      8 from .request import Request

File /opt/homebrew/lib/python3.10/site-packages/specio/core/format.py:10, in <module>
      7 from __future__ import print_function
      9 import os
---> 10 from collections import Iterable

Create additional reader

Add the following reader:

mzXML / mzML
SP file -> IR images
SPE file -> IR images

Add filename into the metadata

Rename specio.Spectrum.spectrum into specio.Spectrum.amplitude

when I read a .sp file, an unicode decode error pops up.
I look up on stackoverflow here, It says i should change encoding method.
Do you know where can I change the encoding method？
here is the file.

ds-035-1-1.zip

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-74-2ccbb1759542> in <module>()
      1 import specio
----> 2 specread(allfiles[0])

C:\ProgramData\Anaconda3\lib\site-packages\specio\core\functions.py in specread(uri, format, tol_wavelength, **kwargs)
    244     filenames = _validate_filenames(uri)
    245     spectrum = [_get_reader_get_data(f, format, **kwargs)
--> 246                 for f in filenames]
    247     return (_zip_spectrum(spectrum, tol_wavelength) if len(spectrum) > 1
    248             else spectrum[0])

C:\ProgramData\Anaconda3\lib\site-packages\specio\core\functions.py in <listcomp>(.0)
    244     filenames = _validate_filenames(uri)
    245     spectrum = [_get_reader_get_data(f, format, **kwargs)
--> 246                 for f in filenames]
    247     return (_zip_spectrum(spectrum, tol_wavelength) if len(spectrum) > 1
    248             else spectrum[0])

C:\ProgramData\Anaconda3\lib\site-packages\specio\core\functions.py in _get_reader_get_data(uri, format, **kwargs)
    115 def _get_reader_get_data(uri, format, **kwargs):
    116     """Get the reader and the associated data."""
--> 117     reader = get_reader(uri, format, **kwargs)
    118     with reader:
    119         return reader.get_data(index=None)

C:\ProgramData\Anaconda3\lib\site-packages\specio\core\functions.py in get_reader(uri, format, **kwargs)
    108 
    109     # Return its reader object
--> 110     return format.get_reader(request)
    111 
    112 # Spectra

C:\ProgramData\Anaconda3\lib\site-packages\specio\core\format.py in get_reader(self, request)
    111 
    112         """
--> 113         return self.Reader(self, request)
    114 
    115     def can_read(self, request):

C:\ProgramData\Anaconda3\lib\site-packages\specio\core\format.py in __init__(self, format, request)
    163             self._request = request
    164             # Open the reader
--> 165             self._open(**self.request.kwargs.copy())
    166 
    167         @property

C:\ProgramData\Anaconda3\lib\site-packages\specio\plugins\sp.py in _open(self)
    387         def _open(self):
    388             self._fp = self.request.get_file()
--> 389             self._data = self._read_sp(self._fp)
    390 
    391         def _close(self):

C:\ProgramData\Anaconda3\lib\site-packages\specio\plugins\sp.py in _read_sp(sp_file)
    357 
    358             meta.update(_decode_5104(
--> 359                 content[start_byte:start_byte + block_size]))
    360 
    361             start_byte = NBP[1]

C:\ProgramData\Anaconda3\lib\site-packages\specio\plugins\sp.py in _decode_5104(data)
     69                 '<h', data[start_byte:start_byte + 2])[0]
     70             start_byte += 2
---> 71             text.append(data[start_byte:start_byte + text_size].decode('utf8'))
     72             start_byte += text_size
     73             start_byte += 6

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd6 in position 25: invalid continuation byte

remove a /specio/.gitignore

Remove .travis.yml which inside specio

Export Spectrum into CSV

We need to be able to export a specio.core.Spectrum into CSV.

Simplify structure

We might be able to simplify the code by using only functions instead of having two classes read and write

Pip install failing on spc

joris@joris-XPS-13-9350:/media/joris/DATA/CDS/Lypsis/specio$ pip install -e .
Obtaining file:///media/joris/DATA/CDS/Lypsis/specio
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/media/joris/DATA/CDS/Lypsis/specio/setup.py", line 2, in <module>
        from specio import __version__
      File "/media/joris/DATA/CDS/Lypsis/specio/specio/__init__.py", line 17, in <module>
        from . import plugins
      File "/media/joris/DATA/CDS/Lypsis/specio/specio/plugins/__init__.py", line 67, in <module>
        from . import spc
      File "/media/joris/DATA/CDS/Lypsis/specio/specio/plugins/spc.py", line 14, in <module>
        import spc
    ImportError: No module named 'spc'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /media/joris/DATA/CDS/Lypsis/specio/

Automatically create the metadata

To avoid fluctuation in the file format, we could make a copy of the available attributes instead of specifying which attributes to keep

Factorize test for all reader

It should be possible to make common tests for the different format:

read one file
read multiple files

Fix warning when compiling the documentation

Read list of 1d spectra

Read 1d spectra with spc format

Simplify the reader base class

We don't need to have the following function to implement in the children classes:

_get_length
_get_meta_data

Create fetcher

For the toy data which are large, we need to create some fetcher to be able to create and upload wheel in PyPi. The git repo will also be lighter

include spc

we can vendor the spc package

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xef

Hello,

Thank you very much for this python library. It is very useful.

However sometimes, depending of the .sp file, I get an unicode decode error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xef in position 36: invalid continuation byte

Do you have any idea how to solve this problem ?
The encoding of my sp file seems to be ANSI.
I have tried to convert it into UTF-8, but it didn't solve the problem.

Thanks in advance.

Originally posted by @ccholet in #53 (comment)

Add better examples.
Add type out.
More detail about Spectrum (example how to use it).
Add more example in the docstring.