Bug report

Bug summary

The function typhon.plots.center_colorbar uses set_clim, which is deprecated since Matplotlib 3.1 and will be removed in 3.3

Code for reproduction

The following code triggers the deprecation warning.

import numpy as np
import matplotilb.pyplot as plt
import typhon as ty


fig, ax = plt.subplots()
sm = ax.pcolormesh(np.random.randn(20, 20) + 1, cmap='seismic')
cb = fig.colorbar(sm)
ty.plots.center_colorbar(cb)

Version information

• Operating System: //
• Typhon Version: master
• Python Version: 3.7

Dear all,

our copyright is inconsistent between the PyPi entry (1990-2017) and our documentation (2015-2016). Especially the latter is a bit outdated 😉

Does anyone really know how this is properly handled?

Cheers,
Lukas

Bug report

Bug summary

typhon.plots.profile_z shows counter-intuitive behavior as the data is implicitly converted form [m] to [km]. This is inconvenient when further modifying the plot.

Code for reproduction

import numpy as np
import matplotlib.pyplot as plt
import typhon

z = np.linspace(0, 80e3, 50)
x = np.sin(z / 5e3)

fig, ax = plt.subplots()
typhon.plots.profile_z(z, x, ax=ax)
ax.set_ylim(top=np.max(z))

plt.show()

Actual outcome

The y-range is compressed as the data is plotted against the scaled coordinates im [km].

Expected outcome

It is more intuitive to plot against the actual coordinates in [m]. The y-Labels could still be shown in [km] for visual appealing (cp. typhon.plots.profile_p_log)

Version information

• Operating System: Linux and OSX
• Typhon Version: 0.4.0dev
• Python Version: 3.6.2

The fsspec library provides a generic interface to interact with filesystems independently of where they are located. This means users can automatically search for files on remote filesystems such as with the s3fs implementation for Amazon S3 buckets, or wrap this in a CachingFileSystem instance caching requested data locally in sparse files. For users who don't have all satellite data available locally the ability to download only the data needed can save significant time.

My specific use case is to find GLM LCFA files covering a specified time period, pass those to glmtools (ideally by passing files opened with a CachingFileSystem instance, but passing files downloaded entirely if needed), so that I can read the resulting gridded products using satpy.

Satpy contains functionality to search for files based on file patterns, similar to what exists in a typhon Fileset, and this functionality supports fsspec FileSystem instances; however, it only supports sensors for which Satpy has a reading routine (Satpy reads gridded GLM, but not ungridded GLM LCFA). It also has ongoing work for generic filesystem support, but this will not help for GLM as long as glmtools interfacing happens outside of satpy (and the latter does not support fsspec filesystems).

Therefore, it would be very nice if the user could pass any instance implementing fsspec.spec.AbstractFileSystem, so that the FileSet functionality is available regardless of where the files are located.

The current handling of environment variables/configuration does not consider changes in environment variables. This is a direct consequence of the implementation as module constants.

I will re-implement the typhon.environment module around a dict-like wrapper class (e.g. os.environ). Within the __getitem__ method both, the config and the environment variables will be considered.

Current implementation

from typhon.environment import ARTS_BUILD_PATH

Future implementation

from typhon.environment import environ

environ['ARTS_BUILDPATH']

To-do list

• Re-implement the typhon.environment handling.
• Replace all occurrences of the old constants with a call to environ
• Add tests to cover possible clashes properly

Cheers,
Lukas

Typhon currently has two competing Dataset approaches. One lives in the typhon.datasets package and its subpackages, another in typhon.spareice.datasets. The two have overlapping aims but diverge in their implementation and approach. This duplicates maintenance effort and in the long term, it is desirable to merge them. This requires a major effort.

To achieve this goal, it would be needed to make a full inventory of the features within each approach, where they overlap and where they differ. I suspect the typhon.datasets package has many features not currently available in typhon.spareice.datasets. It would affect readers implemented in either way.

Limitations of the typhon.datasets package:

• Limited documentation.
• Lacking tests.
• Overuse of multiple inheritance.
• Some methods defined in wrong place in class hierarchy.
• Hard to use. This is related to the limited documentation and the overuse of multiple inheritance, but that is not the only problem.

I don't expect to be able to do much on this any time soon, nor do I expect others to do so, but I'm keeping this issue here so we keep it in mind long term.

Nose has been in maintenance mode for the past several years and will likely cease without a new person/team to take over maintainership. New projects should consider using Nose2, py.test, or just plain unittest/unittest2. - Note to users

As the Python community is vibrant, some projects rise (like typhon 😉) and others decay. It seems like we backed the wrong horse when choosing nose for our tests.

Fortunately, almost all our tests are already compatible with pytest, our dedicated successor of nose. I will migrate the incompatible tests in the next days/weeks in a feature branch and merge them in one piece. It should be easier, if the transition happens at one go.

In addition, pytest offers a lot of additional features like specialized test fixtures and the possibility to run tests in parallel. So we will most likely benefit from the new setup.

Cheers,
Lukas

If I import typhon on a machine where I do not have a X11 connection, I get an error. This results from importing matplotlib which default backend requires a X11 server. The workaround is to import matplotlib before typhon and to set another backend manually. New users or beginners could struggle here, hence it should be covered in a FAQ.

What do you think?

Hey everyone,

I would like to remove the right and top spines from the typhon stylesheet by default.

Before:

After:

The idea is to follow Edward Tufte's concept of an increased data-ink-ratio.

Before merging this change I wanted to get you involved:

• Do you use the stylesheet?
• Do you have a strong argument for keeping the spines?
• Do you want to propose other changes to the default plot layout?

Cheers,
Lukas

Bug report

Bug summary

spareice/test_datasets.py is failing.

Code for reproduction

py.test -v typhon/tests/spareice/test_datasets.py

Actual outcome

E       assert [('typhon/tes... 6, 0))), ...] == [('typhon/test... 6, 0))), ...]
E         At index 0 diff: ('typhon/tests/spareice/reference/2017/01/01/120000-180000.nc', (datetime.datetime(2017, 1, 1, 12, 0), datetime.datetime(2017, 1, 1, 12, 0))) != ('typhon/tests/spareice/reference/2017/01/01/000000-060000.nc', (datetime.datetime(2017, 1, 1, 0, 0), datetime.datetime(2017, 1, 1, 0, 0)))
E         Full diff:
E         - [('typhon/tests/spareice/reference/2017/01/01/120000-180000.nc',
E         ?                                               ^^     ^^
E         + [('typhon/tests/spareice/reference/2017/01/01/000000-060000.nc',
E         ?                                               ^^     ^^
E         -   (datetime.datetime(2017, 1, 1, 12, 0), datetime.datetime(2017, 1, 1, 12, 0))),
E         ?                                  ^^                                    ^^
E         +   (datetime.datetime(2017, 1, 1, 0, 0), datetime.datetime(2017, 1, 1, 0, 0))),
E         Detailed information truncated (25 more lines), use "-vv" to show

Expected outcome

No failure.

Bug report

I got an error when I call cartposlos2geocentric function on geodesy.py

Bug summary

I use atmlab for my research but when I try to do the same in Typhon I got the error

Code for reproduction

My script requires many own variables but I got the same error when I call the function with this parameters

#  cartposlos2geocentric(1,1,1,1,1,1)
#
#

Actual outcome

 IndexError                                Traceback (most recent call last)
<ipython-input-2-9f391afb5afc> in <module>()
----> 1 cartposlos2geocentric(1,1,1,1,1,1)

~/anaconda3/lib/python3.6/site-packages/typhon/geodesy.py in cartposlos2geocentric(x, y, z, dx, dy, dz, ppc, lat0, lon0, za0, aa0)
    702 
    703         fix = np.logical_or(np.isnan(aa), ~np.isreal(aa))
--> 704         aa[np.logical_and(fix, dlat >= 0)] = 0
    705         aa[np.logical_and(fix, dlat < 0)] = 180
    706 

Expected outcome

I expect to receive the same result that I've got from atmlab

Version information

• Operating System: Ubuntu 16.04
• Typhon Version: 0.5.0
• Python Version: 3.6.3

I installed Typhon with pip

I don't know if this is the right way to solve the problem but changing aa[np.logical_and(fix, dlat >= 0)] = 0 for aa[np.bool(np.logical_and(fix, dlat >= 0))] = 0 I've got the right result

I hope I've explained the problem properly

Kind regards,

Federico Cutraro

If the typhon version is not up-to-date with the ARTS C API weird and silent crashes occur. This should be fixed by introducing a check for the ARTS version number on typhon side.

Hi all,

I find myself stumbling across the structure of the tests more and more frequently.

The idea behind the structure as is was to keep implementation and test close to each other in the filetree. In practice, it is often a hassle to determine if there are existing tests and where to find them.

• I suggest to move all tests into the top-level typhon/tests/ directory.

Every subpackage gets an own test module:

tests/
|-- test_atmosphere.py
|-- test_geographical.py
|-- test_latex.py
|-- test_spectroscopy.py
-- test_workspace.py

The structure within the sub-packages could be emulated by classes (TestXML, TestXMLMatpack, ...).

What are your opinions? Any arguments to stick to the old system? Or has anyone an even better idea how to structure our tests in general?

Cheers,
Lukas

Currently an exception is thrown when a variable with the same name is create twice. This is annoying when using notebooks, and should be allowed when the group is correct.

Bug report

Bug summary

Trying to import typhon.arts.workspace with the latest arts-2-3-900 / typhon-0ff43fa leads to the following error:

Loading ARTS API from: /Users/u237002/Hacking/arts-build/gcc/src/libarts_api.so
[snip]
AttributeError: dlsym(0x7fa7d1b07740, set_parameters): symbol not found

Cause of the problem

In arts-2-3-898 the function set_parameters was removed but calls to this function are still present in typhon/arts/workspace/api.py.

See above.

This issue is a simple reminder for myself.

Starting in 2016, Basemap came under new management. The Cartopy project will replace Basemap, but it hasn’t yet implemented all of Basemap’s features. All new software development should try to use Cartopy whenever possible, and existing software should start the process of switching over to use Cartopy. All maintenance and development efforts should be focused on Cartopy. - https://matplotlib.org/basemap/users/intro.html

The examples for typhon.plots.cm are based on Basemap and should be rewritten with cartopy.

Hi,

We have a study coming up which uses the new QRNN implementation and it would be great to have a release for that.
Are there any plans for a soon-ish 0.9 release?

Currently the interactive interface cannot handle infinity as a default argument, this should be fixed.
For example:

ws.OEM(method="li_cg",
       max_iter=40,
       display_progress=1,
       lm_ga_settings= np.array([100.0,5.0,5.0,100.0,5.0,0.0]))

gives

Exception: Generic input max_start_cost expected to be of type Numeric

Bug report

Bug summary

typhon tries to import the module typhon.nonlte.version which does not exist. The problem is in commit 84c0442

Code for reproduction

import typhon

Actual outcome

Traceback (most recent call last):
  File "/dev/shm/u237002/anaconda36/envs/python36/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2910, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-7a30c66f0630>", line 1, in <module>
    import typhon
  File "/scratch/uni/u237/users/jmrziglod/projects/typhon/typhon/__init__.py", line 20, in <module>
    from . import nonlte
  File "/scratch/uni/u237/users/jmrziglod/projects/typhon/typhon/nonlte/__init__.py", line 4, in <module>
    from .version import __version__
ModuleNotFoundError: No module named 'typhon.nonlte.version'

scipy 1.0.0 is out soon. typhon.math.stats.corrcoef has a dependency on scipy.stats.betai. This function is deprecated and due to be removed in scipy 1.0.0. Therefore, typhon should be using scipy.special.betainc instead.

In FileSet, the finest temporal placeholder is the second. GOES-16 and GOES-17 filenames contain a sub-second start and end time for each file (see GOES-R Product Definition and User's Guide, Appendix A, Table A-1, page 606, available here as PDF):

sYYYYDDDHHMMSSs
(...)
SSs = second of minute: 00-60 (... third "s" is tenth of second)

The most general solution would be to allow seconds with any SI-prefix, so that the user could supply {decisecond} and {end_decisecond}.

Bug report

Bug summary

Importing of typhon.retrieval module fails

Code for reproduction

import typhon.retrieval

Actual outcome

ImportError                               Traceback (most recent call last)
<ipython-input-1-1bc9d9b9f8dc> in <module>
----> 1 import typhon.retrieval

~/src/typhon/typhon/retrieval/__init__.py in <module>
      6 from .common import *  # noqa
      7 from .oem import *  # noqa
----> 8 from .spareice import *  # noqa

~/src/typhon/typhon/retrieval/spareice/__init__.py in <module>
      8 """
      9 
---> 10 from .common import *
     11 
     12 __all__ = [s for s in dir() if not s.startswith('_')]

~/src/typhon/typhon/retrieval/spareice/common.py in <module>
     75 from typhon.collocations import collapse, Collocations, Collocator
     76 from typhon.geographical import sea_mask
---> 77 from typhon.plots import binned_statistic, heatmap, styles, worldmap
     78 from typhon.utils import to_array, Timer
     79 import xarray as xr

ImportError: cannot import name 'worldmap' from 'typhon.plots' (/home/simonpf/src/typhon/typhon/plots/__init__.py)

Version information

In [3]: typhon.__version__                                                                                                                                                                                                                    
Out[3]: '0.8.0dev'

Test currently fails with:

typhon/__init__.py:18: in <module>
    from . import geographical
typhon/geographical.py:7: in <module>
    import imageio
E   ModuleNotFoundError: No module named 'imageio'

typhon/geographical.py and typhon/retrieval/spareice/common.py import imageio which is not a dependency of typhon yet. Should we do a late import at the two places where it is used or make it a hard dependency?

Currently stdout is not visible in IPython notebooks, which can be annoying. One could also think about adding some HTML markup.

The last coefficient on line 1524 misses an E.

Bug report

C API does not include the arts-xml-data path in cmake. .typhonrc also seems to be ignored. I have tried using both ARTS_DATA_PATH and ARTS_XML_PATH in .typhonrc but neither works.

Bug summary

Cannot easily read files from arts-xml-data the same way as when using controlfiles when using C API

Code for reproduction

ARTS built with cmake arts-xml-data path defined.

Full .typhonrc:
[environment]
ARTS_BUILD_PATH: path-to-arts-executable-folder
ARTS_DATA_PATH: path-to-arts-xml-data-folder

In script:
from typhon.arts.workspace import Workspace
ws = Workspace()
ws.ReadXML(ws.isotopologue_ratios, "planets/Mars/isotopratio_Mars.xml")

Actual outcome

Exception: Call to ARTS WSM ReadXML failed with error: Cannot find input file: planets/Mars/isotopratio_Mars.xml
Search path: ARTS_PATH HOME_PATH

Expected outcome

I expect that the search path should include ARTS_DATA_PATH and ARTS_XML_PATH. Or at the very least, the arts-xml-data path that is used in cmake when building ARTS.

Version information

• Python Version: latest Anaconda running via graphical spyder3 on ubuntu

Hopeful feature addition

I write this because I suspect I know what the solution will have to look at, without actually knowing how to solve it...

It would be very nice to be able to call Workspace() with tuples like ('D', "data/") and ("n", 17) [or whatever input format is preferred, I do not really care on that point] to respectively add the data import path of 'data/' and to limit the number of threads to 17. (Especially the latter would be nice since leaving a few threads open makes other things on the computer run more smoothly.)

Bug report

Bug summary

#300 reverts commit c2655f6.

Arrays with more than 127 elements were partially uninitialized after
that commit.

Code for reproduction

Test on linux:

import typhon
from typhon.arts.workspace import Workspace
import numpy as np
arts = Workspace(3, 3)
arts.f_grid = np.linspace(1.75e13, 4e14, 128)
print(arts.f_grid.value[0:6])

Actual outcome

Before revert:

[6.90303235e-310 6.90303235e-310 3.17051490e-316 3.17051490e-316
2.95472441e+013 3.25590551e+013]

Expected outcome

After revert:

[1.75000000e+13 2.05118110e+13 2.35236220e+13 2.65354331e+13
2.95472441e+13 3.25590551e+13]

[from @gerritholl]

If I understand generate_filename correctly, the typhon.spareice.datasets approach assumes that the filename can be calculated using only the placeholders in the template. This is not the case for most real datasets. For example, many filenames contain orbit numbers or the string of a downlink station. That means it is necessary to include a regular expression. I'm not sure this is possible with the typhon.spareice.datasets approach but if it isn't, that would be a major limitation.

You are right. So far generate_filename only uses temporal placeholders. I thought about implementing user-defined placeholders but I have not had the time to do it. What do you need them for? Do you want to keep the information from the original filenames and create new filenames with it? A kind of filename conversion? Could you give me a more detailed example? How do you use typhon.Datasets for this?

It would be nice to extend Sphinx' cross-referencing to consider the ARTS built-in documentation browser.

Maybe on can make use of the built-in inline markup and add a new role arts for convenient usage in rst-files and docstrings:

:arts:`atm_fields_compact`

Cheers,
Lukas

Owing to the recent development of these classes, there are just a few tests working and most of them are obsolete. Hence, they need a review and many tests must be added. This issue acts as a ToDo list for tests and will be updated in future.

FileSet and FileHandler

• Test FileSet.align
• Test MHS_HDF.read
• Test AVHRR_HDF.read
• Test CloudSat.read

Collocator, Collocations and related functions

• Test Collocator.collocate
• Test Collocator.collocate_filesets or Collocations.search
• Test collapse
• Test expand

SPAREICE

• Test SPAREICE.retrieve
• Test SPAREICE.retrieve_from_collocations

Bug report

Bug report for Freddy.

The ARTS API cannot handle doit_i_field. Probably because of a missing dimension.

Bug summary

Code for reproduction

import numpy as np
from typhon.arts.workspace import Workspace#, arts_agenda

ws=Workspace()
ws.verbositySetScreen(level=2)

# Dimension / type of atmosphere
ws.AtmosphereSet1D()

# Dimension of Stokes vector
StokesDim=1
ws.IndexSet( ws.stokes_dim, StokesDim )

za_grid=np.linspace(0,180,16)
aa_grid=np.linspace(0,360,6)

ws.VectorSet(ws.scat_za_grid, za_grid)
ws.VectorSet(ws.scat_aa_grid, aa_grid)
ws.IndexSet(ws.doit_za_grid_size,16)
ws.IndexSet(ws.cloudbox_on,1)
ws.ArrayOfIndexSet(ws.cloudbox_limits,[0,1])

ws.DoitInit()
ws.doit_i_field.value.shape

Actual outcome

Loading ARTS API from: /Users/u300509/Documents/arts/build/src/libarts_api.so
- verbosityInit
- verbositySetScreen
- AtmosphereSet1D
  Sets the atmospheric dimensionality to 1.
- DoitInit
Traceback (most recent call last):
  File "doit_i_field_test.py", line 35, in <module>
    ws.doit_i_field.value.shape
  File "/Users/u300509/Documents/typhon/typhon/arts/workspace/workspace.py", line 294, in __getattr__
    wsv = WorkspaceVariable(i, name, group_names[group_id], description, self)
  File "/Users/u300509/Documents/typhon/typhon/arts/workspace/variables.py", line 57, in __init__
    self.update()
  File "/Users/u300509/Documents/typhon/typhon/arts/workspace/variables.py", line 275, in update
    shape.append(v.dimensions[i])
IndexError: invalid index

Version information

• Operating System: Cross-platform
• Typhon Version: Latest github master
• Python Version: 3.6.5

Cheers,
Lukas

[from @gerritholl]

Personally, I would like to see something fully built around xarray.DataArray and xarray.Dataset (it is unfortunate that our class is also called Dataset). I wasn't aware of xarray when I first started the typhon.datasets package. The typhon.spareice.datasets approach uses the ArrayGroup class which appears very similar to an xarray.Dataset, whereas the Array appears very similar to xarray.DataArray, yet they aren't built around it.

I started to build spareice.datasets on xarray but I ran into some trouble hence I wrote my own Array and ArrayGroup implementations. Maybe you have some suggestions for me how to solve them. Here are my thoughts about xarray:

• xarray.Dataset does not have a native group support as we know it from netCDF4 and there is no development on that topic at the moment (afaik from this thread: pydata/xarray#1092). This is a huge drawback for me since I use the group support extensively especially in the collocations tool. It makes it easier to combine data coming from different datasets. This was the main reason for implementing ArrayGroup which allows grouping by using unix-like paths.
• I really like xarray for working with labeled data but when doing elementwise computations it is starting to become a mess due to the label alignment of the underlying pandas (https://pandas.pydata.org/pandas-docs/stable/dsintro.html#vectorized-operations-and-label-alignment-with-series). Hence I preferred to use plain but simple numpy.arrays that are very straightforward to handle.
• Extending xarray should be done via accessors: the developer discourage to subclass xarray directly and suggest writing accessors (http://xarray.pydata.org/en/stable/internals.html#extending-xarray) which follow the composition-over-inheritance principle (well, in general that is a good thing :-) ). But it hold me off to dig deeper into extending xarrays.

Nevertheless xarray has a great dask support which makes it preferable for big data applications and it seems to have a bright future. So I totally agree with you that future specific datasets implementations should support xarray objects. I therefore try to make ArrayGroup compatible to xarray objects. But in general, I think the actual Dataset base class should be independent from its file content - therefore it should not care about xarrays, ArrayGroups or whatever. This makes it more powerful and also usable for datasets of other data types (e.g. text based or images).

What do you think?

Bug report

All tests fail due to not set arts build path.

arts/__init__.py:7: in <module>
    from typhon.arts import xml  # noqa
arts/xml/__init__.py:14: in <module>
    from . import read
arts/xml/read.py:15: in <module>
    from .. import types
arts/types.py:13: in <module>
    from .covariancematrix import (CovarianceMatrix)
arts/covariancematrix.py:4: in <module>
    from typhon.arts.workspace.api import arts_api
arts/workspace/api.py:44: in <module>
    + " locate ARTS API.")
E   OSError: ARTS_BUILD_PATH environment variable required to locate ARTS API.

When looking at the daily build documentation for typhon, the documentation extraction for typhon.trees does not seem to work: http://radiativetransfer.org/misc/typhon/doc-trunk/typhon.trees.html

The class IntervalTree is not clickable and one cannot see the docstrings of the class nor its methods.

Does anybody know why?

import numpy as np
from typhon.physics.units.common import ureg
from typhon.physics.units.em import SRF
frequency = ureg.Quantity(np.array(
[37011414567901.24, 36560055853658.54, 36119573253012.05, 35689578333333.336,
35269700941176.47, 34859588139534.887, 34458903218390.805, 34067324772727.27,
33684545842696.633, 33310273111111.11, 32944226153846.156]), 'Hz')
weight = np.array([1]*11)
srf = SRF(frequency, weight)
print(srf.centroid())

--> 34906834926126.227 hertz ** 2

Why hertz ^2 and not hertz ?

Thanks

Bug report

Bug summary

Access to an uninitialized user created variable is not caught by the API.

Code for reproduction

from typhon.arts.workspace import Workspace
# from arts.workspace import Workspace

ws = Workspace()
ws.IndexCreate("myindex")

print("1st try:")
print(ws.myindex.value)
ws.Print(ws.myindex, 0)

print("2nd try:")
ws = Workspace()
print(ws.myindex.value)
ws.Print(ws.myindex, 0)

Actual outcome

1st try:
8070450532247928832
- Print
8070450532247928832
2nd try:
Traceback (most recent call last):
  File "./uninitialized-user-var.py", line 16, in <module>
    print(ws.myindex.value)
  File "/Users/u237002/Hacking/git/typhon/typhon/arts/workspace/variables.py", line 281, in value
    raise Exception("WorkspaceVariable " + self.name + " is uninitialized.")
Exception: WorkspaceVariable myindex is uninitialized.

Expected outcome

Already on the 1st try, the Exception should be thrown.

Version information

• Operating System: macOS
• Typhon Version: 7ebf429
• Python Version: 3.7.6

It would be nice to have a frequently asked questions (FAQ) section in the typhon documentation.

Purpose

The FAQ section could be used as an application-oriented collection of examples. First ideas would be:

• How do I read and write XML files?
• How do I create an atm_fields_compact?
• How to get started with the ARTS C API?
• ...

Implementation

The implementation should be easily extendable and as automatic as possible. Maybe one can simply add a directory faq (in doc) which could store a single rst-file for each question.

Ideally, these files are included into the documentation without having to update an index by hand.

I will have a look at this at some point. But if there are any Sphinx experts around, I am open to suggestions 😉

Cheers,
Lukas

typhon.datasets

Limitations

• Limited documentation.
• Lacking tests.
• Overuse of multiple inheritance.
• Some methods defined in wrong place in class hierarchy.
• Hard to use. This is related to the limited documentation and the overuse of multiple inheritance, but that is not the only problem.

Features

TODO: Edit to add feature description

typhon.spareice.datasets

Limitations

• No RegEx support for parsing additional information from filenames
• No RegEx support for generating new files
• So far only file handlers for the HDF versions of MHS and CloudSat files are available
• No testing for more complex datasets

Features

• Focus on usability and maintainability
• Should work with any kind of temporal datasets
• Extendable by using customized file handlers
• Mapping functions with multiprocessing support
• find_files method can return files in bundles (either by size or temporal bins)

Most tutorials need an update or to be finished. The documentation strings of many classes (especially SPAREICE) must be reviewed and adapted to the new API.

FileSet

• Finish tutorial

Collocations

• Update + Finish tutorial
• Update documentation strings and examples

SPAREICE

• Add tutorial
• Update documentation strings and examples

Setup AppVeyor project to automatically build and test typhon on a Windows machine.

Bug report

Bug summary

Using conda, I cannot install typhon for Python 3.8. In fact, conda throws an exception. Probably a bug in conda, but possibly triggered by something in typhon as it's not happening with Python 3.7 and I've not seen this bug with any other package installed via conda. The failure appears similar to conda/conda#8811 except that it's supposed to be fixed in conda 4.8.2.

Code for reproduction

conda create -n test -c rttools -c conda-forge --strict-channel-priority python=3.8 typhon

Actual outcome

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... 
Building graph of deps:   0%|          | 0/4 [00:00<?, ?it/s]
Examining @/linux-64::__glibc==2.26=0:   0%|          | 0/4 [00:00<?, ?it/s]
Examining typhon:  25%|██▌       | 1/4 [00:00<00:00, 7724.32it/s]           
Examining python=3.8:  50%|█████     | 2/4 [00:02<00:02,  1.40s/it]
Examining python=3.8:  75%|███████▌  | 3/4 [00:02<00:00,  1.07it/s]
Examining @/linux-64::__cuda==10.0=0:  75%|███████▌  | 3/4 [00:02<00:00,  1.07it/s]
                                                                                   

Determining conflicts:   0%|          | 0/4 [00:00<?, ?it/s]
Examining conflict for typhon python:   0%|          | 0/4 [00:00<?, ?it/s]
Examining conflict for __cuda:  25%|██▌       | 1/4 [00:00<00:02,  1.17it/s]
Examining conflict for __cuda:  50%|█████     | 2/4 [00:00<00:00,  2.34it/s]
                                                                            
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/exceptions.py", line 1079, in __call__
        return func(*args, **kwargs)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/cli/main.py", line 84, in _main
        exit_code = do_call(args, p)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/cli/conda_argparse.py", line 82, in do_call
        return getattr(module, func_name)(args, parser)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/cli/main_create.py", line 37, in execute
        install(args, parser, 'create')
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/cli/install.py", line 261, in install
        unlink_link_transaction = solver.solve_for_transaction(
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 114, in solve_for_transaction
        unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 157, in solve_for_diff
        final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 281, in solve_final_state
        ssc = self._run_sat(ssc)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/common/io.py", line 88, in decorated
        return f(*args, **kwds)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 805, in _run_sat
        ssc.solution_precs = ssc.r.solve(tuple(final_environment_specs),
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/common/io.py", line 88, in decorated
        return f(*args, **kwds)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/resolve.py", line 1318, in solve
        self.find_conflicts(specs, specs_to_add, history_specs)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/resolve.py", line 347, in find_conflicts
        bad_deps = self.build_conflict_map(specs, specs_to_add, history_specs)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/resolve.py", line 506, in build_conflict_map
        c = self.breadth_first_search_for_dep_graph(
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/resolve.py", line 369, in breadth_first_search_for_dep_graph
        last_spec = MatchSpec.union((path[-1], target_paths[-1][-1]))[0]
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 481, in union
        return cls.merge(match_specs, union=True)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 475, in merge
        reduce(lambda x, y: x._merge(y, union), group) if len(group) > 1 else group[0]
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 475, in <lambda>
        reduce(lambda x, y: x._merge(y, union), group) if len(group) > 1 else group[0]
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 502, in _merge
        final = this_component.union(that_component)
      File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 764, in union
        return '|'.join(options)
    TypeError: sequence item 0: expected str instance, Channel found

`$ /data/gholl/miniconda3/bin/conda create -n test -c rttools -c conda-forge --strict-channel-priority python=3.8 typhon`

  environment variables:
         ALSA_CONFIG_PATH=/etc/alsa-pulse.conf
                 CIO_TEST=<not set>
        CONDA_DEFAULT_ENV=/media/nas/x21324/miniconda3
                CONDA_EXE=/data/gholl/miniconda3/bin/conda
             CONDA_PREFIX=/media/nas/x21324/miniconda3
    CONDA_PROMPT_MODIFIER=(/media/nas/x21324/miniconda3)
         CONDA_PYTHON_EXE=/data/gholl/miniconda3/bin/python
               CONDA_ROOT=/data/gholl/miniconda3
              CONDA_SHLVL=1
                  MANPATH=/media/nas/x21308/inst/texlive/2019/texmf-
                          dist/doc/man:/media/nas/x21308/inst/texlive/2019/texmf-
                          dist/doc/man:/media/nas/x21308/inst/texlive/2019/texmf-
                          dist/doc/man:/usr/local/man:/usr/share/man
                 NO_PROXY=<set>
                     PATH=/media/nas/x21308/inst/texlive/2019/bin/x86_64-linux:/media/nas/x21308
                          /inst/texlive/2019/bin/x86_64-linux:/media/nas/x21308/inst/texlive/201
                          9/bin/x86_64-linux:/media/nas/x21324/miniconda3/bin:/media/nas/x21324/
                          miniconda3/condabin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/loc
                          al/bin:/usr/bin:/bin:/opt/bin:/usr/lib/mit/sbin:/opt/perforce/bin:/opt
                          /hdf-java/bin:/usr/lib/mit/sbin:/opt/perforce/bin
            PYTHONSTARTUP=/home/gholl/.pythonrc.py
       REQUESTS_CA_BUNDLE=<not set>
              SOCKS_PROXY=<set>
            SSL_CERT_FILE=<not set>
               WINDOWPATH=7
                 XNLSPATH=/usr/share/X11/nls
                ftp_proxy=<set>
             gopher_proxy=<set>
               http_proxy=<set>
              https_proxy=<set>
                 no_proxy=<set>
              socks_proxy=<set>

     active environment : /media/nas/x21324/miniconda3
    active env location : /media/nas/x21324/miniconda3
            shell level : 1
       user config file : /home/gholl/.condarc
 populated config files : /home/gholl/.condarc
          conda version : 4.8.2
    conda-build version : not installed
         python version : 3.8.1.final.0
       virtual packages : __cuda=10.0
                          __glibc=2.26
       base environment : /data/gholl/miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/rttools/linux-64
                          https://conda.anaconda.org/rttools/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /data/gholl/miniconda3/pkgs
                          /home/gholl/.conda/pkgs
       envs directories : /data/gholl/miniconda3/envs
                          /home/gholl/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.2 requests/2.22.0 CPython/3.8.1 Linux/4.12.14-lp150.12.82-default opensuse-leap/15.0 glibc/2.26
                UID:GID : 16209:16217
             netrc file : None
           offline mode : False


An unexpected error has occurred. Conda has prepared the above report.

If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers?

[y/N]: 
Timeout reached. No report sent.

Expected outcome

I expected a working conda environment with the latest released typhon and Python 3.8.1.

Version information

• Operating System: openSUSE 15.0
• Typhon Version: latest on conda (attempted)
• Python Version: 3.8.1 (attempted)
• Conda Version: 4.8.2

Bug report

Bug summary

test_compress_decompress_zip fails sometimes but not always.

Code for reproduction

from typhon.tests.files.test_utils import TestCompression

test = TestCompression()
passed = 0
for _ in range(50):
    try:
        test.test_compress_decompress_zip()
        passed += 1
    except Exception as error:
        print(f"Passed {passed} times before failed:")
        raise error

Actual outcome

Passed 13 times before failed:
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
typhon/typhon/tests/files/test_utils.py in test_compress_decompress_zip(self)
     22                 self.create_file(compressed_file)
     23 
---> 24             with decompress(file.name+".zip") as uncompressed_file:
     25                 assert self.check_file(uncompressed_file)
     26 

python36/lib/python3.6/contextlib.py in __enter__(self)
     79     def __enter__(self):
     80         try:
---> 81             return next(self.gen)
     82         except StopIteration:
     83             raise RuntimeError("generator didn't yield") from None

typhon/typhon/files/utils.py in decompress(filename, tmpdir)
    179         try:
    180             if fmt == 'zip':
--> 181                 shutil.copyfileobj(compfile(filename, 'r').open(filebase, 'r'),
    182                                    tmpfile,
    183                                    chunksize)

python36/lib/python3.6/zipfile.py in open(self, name, mode, pwd, force_zip64)
   1350         else:
   1351             # Get info object for name
-> 1352             zinfo = self.getinfo(name)
   1353 
   1354         if mode == 'w':

python3.6/zipfile.py in getinfo(self, name)
   1279         if info is None:
   1280             raise KeyError(
-> 1281                 'There is no item named %r in the archive' % name)
   1282 
   1283         return info

KeyError: "There is no item named 'tmpzu27cagz' in the archive"

Machine: platform linux -- Python 3.6.3, pytest-3.3.0, py-1.5.2, pluggy-0.6.0

If the ARTS_INCLUDE_PATH does not have a trailing delimiter, controlfiles are not found.

I will fix this ;) it is more about testing the issue mechanism

Cheers,
Lukas

Bug report

Bug summary

The times returned by the NetCDF4 FileHandler are wrong.

Code for reproduction

import os
import tempfile
import xarray as xr
import numpy as np
from typhon.files import NetCDF4
fh = NetCDF4()

with tempfile.TemporaryDirectory() as tdir:
    tfile = os.path.join(tdir, "testfile.nc")
    before = xr.Dataset(
            {"a":
                xr.DataArray(
                    np.array(
                        ["2019-02-14T09:00:00", "2019-02-14T09:00:01"],
                        dtype="M8[ns]"))})
    before["a"].encoding = {
            "units": "seconds since 2019-02-14 09:00:00",
            "scale_factor": 0.1}
    fh.write(before, tfile)
    after = fh.read(tfile)
    assert np.array_equal(before["a"], after["a"])

Actual outcome

The assertion fails.

Expected outcome

The assertion should succeed.

Bug report

The documentation is broken. First it demands a theme to be installed that is not installed when "pip install -e ." is run. Second it generates an error as below

Bug summary

Documentation behaves unexpectedly poorly

Code for reproduction

IN BASH IN MAIN TYPHON FOLDER:

cd doc
make html
pip install sphinx_rtd_theme
make html

# Paste your code here
#
#

Actual outcome

Just a bunch of strange plots generated

# If applicable, paste the console output here
#
#

looking for now-outdated files...
Exception occurred:
File "/anaconda3/lib/python3.6/site-packages/sphinx/environment/init.py", line 782, in get_doctree
with open(doctree_filename, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/Dropbox/Software/typhon/doc/_build/doctrees/generated/DataFileError.doctree'
The full traceback has been saved in /tmp/sphinx-err-zqk292vi.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at https://github.com/sphinx-doc/sphinx/issues. Thanks!
Makefile:55: recipe for target 'html' failed
make: *** [html] Error 2

NOTE: Also a lot of errors above here with functions or files missing

Expected outcome

I want to have a .html file generated so I can see the documentation

Version information

• Operating System:
• Typhon Version:
• Python Version:

Bug report

Bug report for Freddy.

Bug summary

The workspace group ArrayOfVector is not supported in the API, yet.

Code for reproduction

from typhon.arts.workspace import Workspace

ws = Workspace()
ws.ArrayOfVectorCreate('aov')
ws.VectorCreate('v')
ws.VectorSetConstant(ws.v, 2, 42.)
ws.Append(ws.aov, ws.v)
ws.aov.value

Actual outcome

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-6-452073056e06> in <module>()
      6 ws.VectorSetConstant(ws.v, 2, 42.)
      7 ws.Append(ws.aov, ws.v)
----> 8 ws.aov.value

~/Documents/typhon/typhon/arts/workspace/variables.py in value(self)
    218                 raise Exception("Variable of type " + self.group + " is empty.")
    219         else:
--> 220             raise Exception("Type of workspace variable is not supported by the interface.")
    221 
    222     def update(self):

Exception: Type of workspace variable is not supported by the interface.

Expected outcome

It would be nice to support ArrayOfVector as it is used for ybatch results.

Version information

• Operating System: osx/Linux
• Typhon Version: 0.4.0dev
• Python Version: 3.6

Bug report

Bug summary

The NetCDF4 FileHandler throws a ValueError when trying to read masked dimensionless values. Those occur in GOES GLM L2 NetCDF files.

Code for reproduction

import xarray
from typhon.files import NetCDF4
ds = xarray.Dataset(
        {"a":
            xarray.DataArray(
                42,
                encoding={"_FillValue": 42})})
ds.to_netcdf("/tmp/spam.nc")
fh = NetCDF4()
ds2 = fh.read("/tmp/spam.nc")

Actual outcome

Traceback (most recent call last):
  File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/variable.py", line 79, in as_variable
    obj = Variable(*obj)
  File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/variable.py", line 259, in __init__
    self._data = as_compatible_data(data, fastpath=fastpath)
  File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/variable.py", line 176, in as_compatible_data
    data[mask] = fill_value
ValueError: assignment destination is read-only

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "mwe.py", line 10, in <module>
    ds2 = fh.read("/tmp/spam.nc")
  File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/typhon/files/handlers/common.py", line 111, in wrapper
    return method(*args, **kwargs)
  File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/typhon/files/handlers/common.py", line 660, in read
    self._load_group(dataset, None, root, fields)
  File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/typhon/files/handlers/common.py", line 718, in _load_group
    ds[path + var_name] = dims, var[:], dict(var.__dict__)
  File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/dataset.py", line 980, in __setitem__
    self.update({key: value})
  File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/dataset.py", line 2505, in update
    variables, coord_names, dims = dataset_update_method(self, other)
  File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/merge.py", line 585, in dataset_update_method
    indexes=dataset.indexes)
  File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/merge.py", line 440, in merge_core
    expanded = expand_variable_dicts(aligned)
  File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/merge.py", line 214, in expand_variable_dicts
    var = as_variable(var, name=name)
  File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/variable.py", line 84, in as_variable
    '{} to Variable.'.format(obj))
ValueError: Could not convert tuple of form (dims, data[, attrs, encoding]): ([], masked, {'_FillValue': 42}) to Variable.

Expected outcome

The expected outcome would be to get an xarray.Dataset object with the contents of the file.

Version information

• Operating System: SUSE Linux Enterprise Server 11 (x86_64), patchlevel 4
• Typhon Version: 0.7.0+dev13.g0c85031 (latest git master at the time of writing)
• Python Version: 3.7.2

I installed most through conda, but llvmlite, numba, and typhon through pip.

If a supergeneric method has implementations for more than two combinations of input/output, the current implementation fails to find anything but the first two combinations. E.g. the following implementations for Append exist:

1. Append(ArrayOfArrayOfIndex, ArrayOfArrayOfIndex)
2. Append(ArrayOfIndex, ArrayOfIndex)
3. Append(ArrayOfArrayOfIndex, ArrayOfIndex)

Trying to call the 3rd form of Append currently fails with the error that such an overload cannot be found despite it existing.

Realised through FCDR_HIRS but the problem lies in typhon. On numpy 1.13:

$ plot_hirs_field_timeseries --plot_noise_with_other --without_gain --without_rself --temp_fields --count_fields space --with_corr above --corr_count 6 --corr_timeres 30min --sat noaa15 --channel 1 --from_date 1999-01-01 --to_date 1999-01-2
WARNING  2017-12-18 11:00:34,532 registry._define_single_adder:324: Redefining 'micro' (<class 
'pint.definitions.PrefixDefinition'>)
WARNING  2017-12-18 11:00:34,532 registry._define_single_adder:324: Redefining 'µ' (<class 'pin
t.definitions.PrefixDefinition'>)
INFO     2017-12-18 11:00:36,346 dataset.read_period:373: Reading fcdr_hirs {'satname': 'noaa15
'} for period 1999-01-01 00:00:00  – 1999-01-02 00:00:00
[                                                                     ] N/A% (ETA:  --:--:--) /
dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/numpy/ma/core.py:3008: FutureWarning
: Numpy has detected that you (may be) writing to an array returned
by numpy.diagonal or by selecting multiple fields in a structured
array. This code will likely break in a future numpy release --
see numpy.diagonal or arrays.indexing reference docs for details.
The quick fix is to make an explicit copy (e.g., do
arr.diagonal().copy() or arr[['f0','f1']].copy()).
  if (obj.__array_interface__["data"][0]
/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/numpy/ma/core.py:3009: FutureWarnin
g: Numpy has detected that you (may be) writing to an array returned
by numpy.diagonal or by selecting multiple fields in a structured
array. This code will likely break in a future numpy release --
see numpy.diagonal or arrays.indexing reference docs for details.
The quick fix is to make an explicit copy (e.g., do
arr.diagonal().copy() or arr[['f0','f1']].copy()).
  != self.__array_interface__["data"][0]):
[===================                                                   ]  27% (ETA:  0:00:11) W
ARNING  2017-12-18 11:00:41,122 filters.filter:151: 1 scanlines are out of order, resorting
WARNING  2017-12-18 11:00:41,132 filters.filter:159: 24 duplicate scanlines (judging from scanl
ine number), removing
[======================================================================] 100% (Time: 0:00:13) 

Continues happily beyond this point. But on numpy 1.14, it fails at this point with:

Traceback (most recent call last):
  File "/dev/shm/gerrit/venv/stable-3.6/bin/plot_hirs_field_timeseries", line 11, in <module>
    load_entry_point('FCDR-HIRS==0.1.15+dev23.g827117e', 'console_scripts', 'plot_hirs_field_ti
meseries')()
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/FCDR_HIRS/analysis/timeseri
es.py", line 1370, in main
    temp_fields=p.temp_fields)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/FCDR_HIRS/analysis/timeseri
es.py", line 337, in __init__
    NO_CACHE=True, **hrsargs)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/utils/cache.py", lin
e 54, in wrapper
    return user_function(*args, **kwds)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/datasets/dataset.py"
, line 477, in read_period
    arr = of.finalise(arr)
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/datasets/filters.py"
, line 509, in finalise
    fields_notclose = {nm for nm in rep.dtype.names
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/datasets/filters.py"
, line 513, in <setcomp>
    else numpy.isclose(rep[nm][0, ...], rep[nm])
  File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/numpy/core/numeric.py", lin
e 2329, in isclose
    dt = multiarray.result_type(y, 1.)
TypeError: invalid type promotion

There may be other places where typhon functionality fails on numpy 1.14. I suspect this is due to me doing something inappropriate to which numpy has become less tolerant, but this remains to be seen.