atmtools / typhon Goto Github PK
View Code? Open in Web Editor NEWTools for atmospheric research
Home Page: http://www.radiativetransfer.org/
License: MIT License
Tools for atmospheric research
Home Page: http://www.radiativetransfer.org/
License: MIT License
The function typhon.plots.center_colorbar
uses set_clim
, which is deprecated since Matplotlib 3.1 and will be removed in 3.3
The following code triggers the deprecation warning.
import numpy as np
import matplotilb.pyplot as plt
import typhon as ty
fig, ax = plt.subplots()
sm = ax.pcolormesh(np.random.randn(20, 20) + 1, cmap='seismic')
cb = fig.colorbar(sm)
ty.plots.center_colorbar(cb)
Dear all,
our copyright is inconsistent between the PyPi entry (1990-2017) and our documentation (2015-2016). Especially the latter is a bit outdated ๐
Does anyone really know how this is properly handled?
Cheers,
Lukas
typhon.plots.profile_z
shows counter-intuitive behavior as the data is implicitly converted form [m] to [km]. This is inconvenient when further modifying the plot.
import numpy as np
import matplotlib.pyplot as plt
import typhon
z = np.linspace(0, 80e3, 50)
x = np.sin(z / 5e3)
fig, ax = plt.subplots()
typhon.plots.profile_z(z, x, ax=ax)
ax.set_ylim(top=np.max(z))
plt.show()
The y-range is compressed as the data is plotted against the scaled coordinates im [km].
It is more intuitive to plot against the actual coordinates in [m]. The y-Labels could still be shown in [km] for visual appealing (cp. typhon.plots.profile_p_log
)
The fsspec library provides a generic interface to interact with filesystems independently of where they are located. This means users can automatically search for files on remote filesystems such as with the s3fs implementation for Amazon S3 buckets, or wrap this in a CachingFileSystem
instance caching requested data locally in sparse files. For users who don't have all satellite data available locally the ability to download only the data needed can save significant time.
My specific use case is to find GLM LCFA files covering a specified time period, pass those to glmtools (ideally by passing files opened with a CachingFileSystem
instance, but passing files downloaded entirely if needed), so that I can read the resulting gridded products using satpy.
Satpy contains functionality to search for files based on file patterns, similar to what exists in a typhon Fileset, and this functionality supports fsspec FileSystem instances; however, it only supports sensors for which Satpy has a reading routine (Satpy reads gridded GLM, but not ungridded GLM LCFA). It also has ongoing work for generic filesystem support, but this will not help for GLM as long as glmtools interfacing happens outside of satpy (and the latter does not support fsspec filesystems).
Therefore, it would be very nice if the user could pass any instance implementing fsspec.spec.AbstractFileSystem
, so that the FileSet
functionality is available regardless of where the files are located.
The current handling of environment variables/configuration does not consider changes in environment variables. This is a direct consequence of the implementation as module constants.
I will re-implement the typhon.environment
module around a dict-like wrapper class (e.g. os.environ
). Within the __getitem__
method both, the config and the environment variables will be considered.
from typhon.environment import ARTS_BUILD_PATH
from typhon.environment import environ
environ['ARTS_BUILDPATH']
typhon.environment
handling.environ
Cheers,
Lukas
Typhon currently has two competing Dataset approaches. One lives in the typhon.datasets package and its subpackages, another in typhon.spareice.datasets. The two have overlapping aims but diverge in their implementation and approach. This duplicates maintenance effort and in the long term, it is desirable to merge them. This requires a major effort.
To achieve this goal, it would be needed to make a full inventory of the features within each approach, where they overlap and where they differ. I suspect the typhon.datasets package has many features not currently available in typhon.spareice.datasets. It would affect readers implemented in either way.
Limitations of the typhon.datasets package:
I don't expect to be able to do much on this any time soon, nor do I expect others to do so, but I'm keeping this issue here so we keep it in mind long term.
Nose has been in maintenance mode for the past several years and will likely cease without a new person/team to take over maintainership. New projects should consider using Nose2, py.test, or just plain unittest/unittest2. - Note to users
As the Python community is vibrant, some projects rise (like typhon ๐) and others decay. It seems like we backed the wrong horse when choosing nose
for our tests.
Fortunately, almost all our tests are already compatible with pytest
, our dedicated successor of nose
. I will migrate the incompatible tests in the next days/weeks in a feature branch and merge them in one piece. It should be easier, if the transition happens at one go.
In addition, pytest
offers a lot of additional features like specialized test fixtures and the possibility to run tests in parallel. So we will most likely benefit from the new setup.
Cheers,
Lukas
If I import typhon on a machine where I do not have a X11 connection, I get an error. This results from importing matplotlib which default backend requires a X11 server. The workaround is to import matplotlib before typhon and to set another backend manually. New users or beginners could struggle here, hence it should be covered in a FAQ.
What do you think?
Hey everyone,
I would like to remove the right and top spines from the typhon stylesheet by default.
The idea is to follow Edward Tufte's concept of an increased data-ink-ratio.
Before merging this change I wanted to get you involved:
Cheers,
Lukas
spareice/test_datasets.py
is failing.
py.test -v typhon/tests/spareice/test_datasets.py
E assert [('typhon/tes... 6, 0))), ...] == [('typhon/test... 6, 0))), ...]
E At index 0 diff: ('typhon/tests/spareice/reference/2017/01/01/120000-180000.nc', (datetime.datetime(2017, 1, 1, 12, 0), datetime.datetime(2017, 1, 1, 12, 0))) != ('typhon/tests/spareice/reference/2017/01/01/000000-060000.nc', (datetime.datetime(2017, 1, 1, 0, 0), datetime.datetime(2017, 1, 1, 0, 0)))
E Full diff:
E - [('typhon/tests/spareice/reference/2017/01/01/120000-180000.nc',
E ? ^^ ^^
E + [('typhon/tests/spareice/reference/2017/01/01/000000-060000.nc',
E ? ^^ ^^
E - (datetime.datetime(2017, 1, 1, 12, 0), datetime.datetime(2017, 1, 1, 12, 0))),
E ? ^^ ^^
E + (datetime.datetime(2017, 1, 1, 0, 0), datetime.datetime(2017, 1, 1, 0, 0))),
E Detailed information truncated (25 more lines), use "-vv" to show
No failure.
I got an error when I call cartposlos2geocentric function on geodesy.py
I use atmlab for my research but when I try to do the same in Typhon I got the error
My script requires many own variables but I got the same error when I call the function with this parameters
# cartposlos2geocentric(1,1,1,1,1,1)
#
#
IndexError Traceback (most recent call last)
<ipython-input-2-9f391afb5afc> in <module>()
----> 1 cartposlos2geocentric(1,1,1,1,1,1)
~/anaconda3/lib/python3.6/site-packages/typhon/geodesy.py in cartposlos2geocentric(x, y, z, dx, dy, dz, ppc, lat0, lon0, za0, aa0)
702
703 fix = np.logical_or(np.isnan(aa), ~np.isreal(aa))
--> 704 aa[np.logical_and(fix, dlat >= 0)] = 0
705 aa[np.logical_and(fix, dlat < 0)] = 180
706
I expect to receive the same result that I've got from atmlab
I installed Typhon with pip
I don't know if this is the right way to solve the problem but changing aa[np.logical_and(fix, dlat >= 0)] = 0 for aa[np.bool(np.logical_and(fix, dlat >= 0))] = 0 I've got the right result
I hope I've explained the problem properly
Kind regards,
Federico Cutraro
If the typhon version is not up-to-date with the ARTS C API weird and silent crashes occur. This should be fixed by introducing a check for the ARTS version number on typhon side.
Hi all,
I find myself stumbling across the structure of the tests more and more frequently.
The idea behind the structure as is was to keep implementation and test close to each other in the filetree. In practice, it is often a hassle to determine if there are existing tests and where to find them.
typhon/tests/
directory.Every subpackage gets an own test module:
tests/
|-- test_atmosphere.py
|-- test_geographical.py
|-- test_latex.py
|-- test_spectroscopy.py
-- test_workspace.py
The structure within the sub-packages could be emulated by classes (TestXML
, TestXMLMatpack
, ...).
What are your opinions? Any arguments to stick to the old system? Or has anyone an even better idea how to structure our tests in general?
Cheers,
Lukas
Currently an exception is thrown when a variable with the same name is create twice. This is annoying when using notebooks, and should be allowed when the group is correct.
Trying to import typhon.arts.workspace
with the latest arts-2-3-900 / typhon-0ff43fa leads to the following error:
Loading ARTS API from: /Users/u237002/Hacking/arts-build/gcc/src/libarts_api.so
[snip]
AttributeError: dlsym(0x7fa7d1b07740, set_parameters): symbol not found
In arts-2-3-898 the function set_parameters was removed but calls to this function are still present in typhon/arts/workspace/api.py.
See above.
This issue is a simple reminder for myself.
Starting in 2016, Basemap came under new management. The Cartopy project will replace Basemap, but it hasnโt yet implemented all of Basemapโs features. All new software development should try to use Cartopy whenever possible, and existing software should start the process of switching over to use Cartopy. All maintenance and development efforts should be focused on Cartopy. - https://matplotlib.org/basemap/users/intro.html
The examples for typhon.plots.cm
are based on Basemap and should be rewritten with cartopy.
Hi,
We have a study coming up which uses the new QRNN implementation and it would be great to have a release for that.
Are there any plans for a soon-ish 0.9 release?
Currently the interactive interface cannot handle infinity as a default argument, this should be fixed.
For example:
ws.OEM(method="li_cg",
max_iter=40,
display_progress=1,
lm_ga_settings= np.array([100.0,5.0,5.0,100.0,5.0,0.0]))
gives
Exception: Generic input max_start_cost expected to be of type Numeric
typhon tries to import the module typhon.nonlte.version which does not exist. The problem is in commit 84c0442
import typhon
Traceback (most recent call last):
File "/dev/shm/u237002/anaconda36/envs/python36/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2910, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-2-7a30c66f0630>", line 1, in <module>
import typhon
File "/scratch/uni/u237/users/jmrziglod/projects/typhon/typhon/__init__.py", line 20, in <module>
from . import nonlte
File "/scratch/uni/u237/users/jmrziglod/projects/typhon/typhon/nonlte/__init__.py", line 4, in <module>
from .version import __version__
ModuleNotFoundError: No module named 'typhon.nonlte.version'
scipy 1.0.0 is out soon. typhon.math.stats.corrcoef has a dependency on scipy.stats.betai. This function is deprecated and due to be removed in scipy 1.0.0. Therefore, typhon should be using scipy.special.betainc instead.
In FileSet
, the finest temporal placeholder is the second
. GOES-16 and GOES-17 filenames contain a sub-second start and end time for each file (see GOES-R Product Definition and User's Guide, Appendix A, Table A-1, page 606, available here as PDF):
sYYYYDDDHHMMSSs
(...)
SSs = second of minute: 00-60 (... third "s" is tenth of second)
The most general solution would be to allow seconds with any SI-prefix, so that the user could supply {decisecond}
and {end_decisecond}
.
Importing of typhon.retrieval
module fails
import typhon.retrieval
ImportError Traceback (most recent call last)
<ipython-input-1-1bc9d9b9f8dc> in <module>
----> 1 import typhon.retrieval
~/src/typhon/typhon/retrieval/__init__.py in <module>
6 from .common import * # noqa
7 from .oem import * # noqa
----> 8 from .spareice import * # noqa
~/src/typhon/typhon/retrieval/spareice/__init__.py in <module>
8 """
9
---> 10 from .common import *
11
12 __all__ = [s for s in dir() if not s.startswith('_')]
~/src/typhon/typhon/retrieval/spareice/common.py in <module>
75 from typhon.collocations import collapse, Collocations, Collocator
76 from typhon.geographical import sea_mask
---> 77 from typhon.plots import binned_statistic, heatmap, styles, worldmap
78 from typhon.utils import to_array, Timer
79 import xarray as xr
ImportError: cannot import name 'worldmap' from 'typhon.plots' (/home/simonpf/src/typhon/typhon/plots/__init__.py)
In [3]: typhon.__version__
Out[3]: '0.8.0dev'
Test currently fails with:
typhon/__init__.py:18: in <module>
from . import geographical
typhon/geographical.py:7: in <module>
import imageio
E ModuleNotFoundError: No module named 'imageio'
typhon/geographical.py
and typhon/retrieval/spareice/common.py
import imageio which is not a dependency of typhon yet. Should we do a late import at the two places where it is used or make it a hard dependency?
Currently stdout is not visible in IPython notebooks, which can be annoying. One could also think about adding some HTML markup.
The last coefficient on line 1524 misses an E.
C API does not include the arts-xml-data path in cmake. .typhonrc also seems to be ignored. I have tried using both ARTS_DATA_PATH and ARTS_XML_PATH in .typhonrc but neither works.
Cannot easily read files from arts-xml-data the same way as when using controlfiles when using C API
ARTS built with cmake arts-xml-data path defined.
Full .typhonrc:
[environment]
ARTS_BUILD_PATH: path-to-arts-executable-folder
ARTS_DATA_PATH: path-to-arts-xml-data-folder
In script:
from typhon.arts.workspace import Workspace
ws = Workspace()
ws.ReadXML(ws.isotopologue_ratios, "planets/Mars/isotopratio_Mars.xml")
Exception: Call to ARTS WSM ReadXML failed with error: Cannot find input file: planets/Mars/isotopratio_Mars.xml
Search path: ARTS_PATH HOME_PATH
I expect that the search path should include ARTS_DATA_PATH and ARTS_XML_PATH. Or at the very least, the arts-xml-data path that is used in cmake when building ARTS.
I write this because I suspect I know what the solution will have to look at, without actually knowing how to solve it...
It would be very nice to be able to call Workspace() with tuples like ('D', "data/") and ("n", 17) [or whatever input format is preferred, I do not really care on that point] to respectively add the data import path of 'data/' and to limit the number of threads to 17. (Especially the latter would be nice since leaving a few threads open makes other things on the computer run more smoothly.)
Arrays with more than 127 elements were partially uninitialized after
that commit.
Test on linux:
import typhon
from typhon.arts.workspace import Workspace
import numpy as np
arts = Workspace(3, 3)
arts.f_grid = np.linspace(1.75e13, 4e14, 128)
print(arts.f_grid.value[0:6])
Before revert:
[6.90303235e-310 6.90303235e-310 3.17051490e-316 3.17051490e-316
2.95472441e+013 3.25590551e+013]
After revert:
[1.75000000e+13 2.05118110e+13 2.35236220e+13 2.65354331e+13
2.95472441e+13 3.25590551e+13]
[from @gerritholl]
If I understand
generate_filename
correctly, the typhon.spareice.datasets approach assumes that the filename can be calculated using only the placeholders in the template. This is not the case for most real datasets. For example, many filenames contain orbit numbers or the string of a downlink station. That means it is necessary to include a regular expression. I'm not sure this is possible with the typhon.spareice.datasets approach but if it isn't, that would be a major limitation.
You are right. So far generate_filename
only uses temporal placeholders. I thought about implementing user-defined placeholders but I have not had the time to do it. What do you need them for? Do you want to keep the information from the original filenames and create new filenames with it? A kind of filename conversion? Could you give me a more detailed example? How do you use typhon.Datasets for this?
It would be nice to extend Sphinx' cross-referencing to consider the ARTS built-in documentation browser.
Maybe on can make use of the built-in inline markup and add a new role arts
for convenient usage in rst-files and docstrings:
:arts:`atm_fields_compact`
Cheers,
Lukas
Owing to the recent development of these classes, there are just a few tests working and most of them are obsolete. Hence, they need a review and many tests must be added. This issue acts as a ToDo list for tests and will be updated in future.
FileSet.align
MHS_HDF.read
AVHRR_HDF.read
CloudSat.read
Collocator.collocate
Collocator.collocate_filesets
or Collocations.search
collapse
expand
SPAREICE.retrieve
SPAREICE.retrieve_from_collocations
Bug report for Freddy.
The ARTS API cannot handle doit_i_field
. Probably because of a missing dimension.
import numpy as np
from typhon.arts.workspace import Workspace#, arts_agenda
ws=Workspace()
ws.verbositySetScreen(level=2)
# Dimension / type of atmosphere
ws.AtmosphereSet1D()
# Dimension of Stokes vector
StokesDim=1
ws.IndexSet( ws.stokes_dim, StokesDim )
za_grid=np.linspace(0,180,16)
aa_grid=np.linspace(0,360,6)
ws.VectorSet(ws.scat_za_grid, za_grid)
ws.VectorSet(ws.scat_aa_grid, aa_grid)
ws.IndexSet(ws.doit_za_grid_size,16)
ws.IndexSet(ws.cloudbox_on,1)
ws.ArrayOfIndexSet(ws.cloudbox_limits,[0,1])
ws.DoitInit()
ws.doit_i_field.value.shape
Loading ARTS API from: /Users/u300509/Documents/arts/build/src/libarts_api.so
- verbosityInit
- verbositySetScreen
- AtmosphereSet1D
Sets the atmospheric dimensionality to 1.
- DoitInit
Traceback (most recent call last):
File "doit_i_field_test.py", line 35, in <module>
ws.doit_i_field.value.shape
File "/Users/u300509/Documents/typhon/typhon/arts/workspace/workspace.py", line 294, in __getattr__
wsv = WorkspaceVariable(i, name, group_names[group_id], description, self)
File "/Users/u300509/Documents/typhon/typhon/arts/workspace/variables.py", line 57, in __init__
self.update()
File "/Users/u300509/Documents/typhon/typhon/arts/workspace/variables.py", line 275, in update
shape.append(v.dimensions[i])
IndexError: invalid index
Cheers,
Lukas
[from @gerritholl]
Personally, I would like to see something fully built around xarray.DataArray and xarray.Dataset (it is unfortunate that our class is also called Dataset). I wasn't aware of xarray when I first started the typhon.datasets package. The typhon.spareice.datasets approach uses the ArrayGroup class which appears very similar to an xarray.Dataset, whereas the Array appears very similar to xarray.DataArray, yet they aren't built around it.
I started to build spareice.datasets on xarray but I ran into some trouble hence I wrote my own Array and ArrayGroup implementations. Maybe you have some suggestions for me how to solve them. Here are my thoughts about xarray:
Nevertheless xarray has a great dask support which makes it preferable for big data applications and it seems to have a bright future. So I totally agree with you that future specific datasets implementations should support xarray objects. I therefore try to make ArrayGroup compatible to xarray objects. But in general, I think the actual Dataset base class should be independent from its file content - therefore it should not care about xarrays, ArrayGroups or whatever. This makes it more powerful and also usable for datasets of other data types (e.g. text based or images).
What do you think?
All tests fail due to not set arts build path.
arts/__init__.py:7: in <module>
from typhon.arts import xml # noqa
arts/xml/__init__.py:14: in <module>
from . import read
arts/xml/read.py:15: in <module>
from .. import types
arts/types.py:13: in <module>
from .covariancematrix import (CovarianceMatrix)
arts/covariancematrix.py:4: in <module>
from typhon.arts.workspace.api import arts_api
arts/workspace/api.py:44: in <module>
+ " locate ARTS API.")
E OSError: ARTS_BUILD_PATH environment variable required to locate ARTS API.
When looking at the daily build documentation for typhon, the documentation extraction for typhon.trees does not seem to work: http://radiativetransfer.org/misc/typhon/doc-trunk/typhon.trees.html
The class IntervalTree is not clickable and one cannot see the docstrings of the class nor its methods.
Does anybody know why?
import numpy as np
from typhon.physics.units.common import ureg
from typhon.physics.units.em import SRF
frequency = ureg.Quantity(np.array(
[37011414567901.24, 36560055853658.54, 36119573253012.05, 35689578333333.336,
35269700941176.47, 34859588139534.887, 34458903218390.805, 34067324772727.27,
33684545842696.633, 33310273111111.11, 32944226153846.156]), 'Hz')
weight = np.array([1]*11)
srf = SRF(frequency, weight)
print(srf.centroid())
--> 34906834926126.227 hertz ** 2
Why hertz ^2 and not hertz ?
Thanks
Access to an uninitialized user created variable is not caught by the API.
from typhon.arts.workspace import Workspace
# from arts.workspace import Workspace
ws = Workspace()
ws.IndexCreate("myindex")
print("1st try:")
print(ws.myindex.value)
ws.Print(ws.myindex, 0)
print("2nd try:")
ws = Workspace()
print(ws.myindex.value)
ws.Print(ws.myindex, 0)
1st try:
8070450532247928832
- Print
8070450532247928832
2nd try:
Traceback (most recent call last):
File "./uninitialized-user-var.py", line 16, in <module>
print(ws.myindex.value)
File "/Users/u237002/Hacking/git/typhon/typhon/arts/workspace/variables.py", line 281, in value
raise Exception("WorkspaceVariable " + self.name + " is uninitialized.")
Exception: WorkspaceVariable myindex is uninitialized.
Already on the 1st try, the Exception should be thrown.
It would be nice to have a frequently asked questions (FAQ) section in the typhon documentation.
The FAQ section could be used as an application-oriented collection of examples. First ideas would be:
atm_fields_compact
?The implementation should be easily extendable and as automatic as possible. Maybe one can simply add a directory faq
(in doc
) which could store a single rst-file for each question.
Ideally, these files are included into the documentation without having to update an index by hand.
I will have a look at this at some point. But if there are any Sphinx experts around, I am open to suggestions ๐
Cheers,
Lukas
TODO: Edit to add feature description
find_files
method can return files in bundles (either by size or temporal bins)Most tutorials need an update or to be finished. The documentation strings of many classes (especially SPAREICE) must be reviewed and adapted to the new API.
Setup AppVeyor project to automatically build and test typhon on a Windows machine.
Using conda, I cannot install typhon for Python 3.8. In fact, conda throws an exception. Probably a bug in conda, but possibly triggered by something in typhon as it's not happening with Python 3.7 and I've not seen this bug with any other package installed via conda. The failure appears similar to conda/conda#8811 except that it's supposed to be fixed in conda 4.8.2.
conda create -n test -c rttools -c conda-forge --strict-channel-priority python=3.8 typhon
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working...
Building graph of deps: 0%| | 0/4 [00:00<?, ?it/s]
Examining @/linux-64::__glibc==2.26=0: 0%| | 0/4 [00:00<?, ?it/s]
Examining typhon: 25%|โโโ | 1/4 [00:00<00:00, 7724.32it/s]
Examining python=3.8: 50%|โโโโโ | 2/4 [00:02<00:02, 1.40s/it]
Examining python=3.8: 75%|โโโโโโโโ | 3/4 [00:02<00:00, 1.07it/s]
Examining @/linux-64::__cuda==10.0=0: 75%|โโโโโโโโ | 3/4 [00:02<00:00, 1.07it/s]
Determining conflicts: 0%| | 0/4 [00:00<?, ?it/s]
Examining conflict for typhon python: 0%| | 0/4 [00:00<?, ?it/s]
Examining conflict for __cuda: 25%|โโโ | 1/4 [00:00<00:02, 1.17it/s]
Examining conflict for __cuda: 50%|โโโโโ | 2/4 [00:00<00:00, 2.34it/s]
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/exceptions.py", line 1079, in __call__
return func(*args, **kwargs)
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/cli/main.py", line 84, in _main
exit_code = do_call(args, p)
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/cli/conda_argparse.py", line 82, in do_call
return getattr(module, func_name)(args, parser)
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/cli/main_create.py", line 37, in execute
install(args, parser, 'create')
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/cli/install.py", line 261, in install
unlink_link_transaction = solver.solve_for_transaction(
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 114, in solve_for_transaction
unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 157, in solve_for_diff
final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 281, in solve_final_state
ssc = self._run_sat(ssc)
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/common/io.py", line 88, in decorated
return f(*args, **kwds)
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/core/solve.py", line 805, in _run_sat
ssc.solution_precs = ssc.r.solve(tuple(final_environment_specs),
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/common/io.py", line 88, in decorated
return f(*args, **kwds)
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/resolve.py", line 1318, in solve
self.find_conflicts(specs, specs_to_add, history_specs)
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/resolve.py", line 347, in find_conflicts
bad_deps = self.build_conflict_map(specs, specs_to_add, history_specs)
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/resolve.py", line 506, in build_conflict_map
c = self.breadth_first_search_for_dep_graph(
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/resolve.py", line 369, in breadth_first_search_for_dep_graph
last_spec = MatchSpec.union((path[-1], target_paths[-1][-1]))[0]
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 481, in union
return cls.merge(match_specs, union=True)
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 475, in merge
reduce(lambda x, y: x._merge(y, union), group) if len(group) > 1 else group[0]
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 475, in <lambda>
reduce(lambda x, y: x._merge(y, union), group) if len(group) > 1 else group[0]
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 502, in _merge
final = this_component.union(that_component)
File "/data/gholl/miniconda3/lib/python3.8/site-packages/conda/models/match_spec.py", line 764, in union
return '|'.join(options)
TypeError: sequence item 0: expected str instance, Channel found
`$ /data/gholl/miniconda3/bin/conda create -n test -c rttools -c conda-forge --strict-channel-priority python=3.8 typhon`
environment variables:
ALSA_CONFIG_PATH=/etc/alsa-pulse.conf
CIO_TEST=<not set>
CONDA_DEFAULT_ENV=/media/nas/x21324/miniconda3
CONDA_EXE=/data/gholl/miniconda3/bin/conda
CONDA_PREFIX=/media/nas/x21324/miniconda3
CONDA_PROMPT_MODIFIER=(/media/nas/x21324/miniconda3)
CONDA_PYTHON_EXE=/data/gholl/miniconda3/bin/python
CONDA_ROOT=/data/gholl/miniconda3
CONDA_SHLVL=1
MANPATH=/media/nas/x21308/inst/texlive/2019/texmf-
dist/doc/man:/media/nas/x21308/inst/texlive/2019/texmf-
dist/doc/man:/media/nas/x21308/inst/texlive/2019/texmf-
dist/doc/man:/usr/local/man:/usr/share/man
NO_PROXY=<set>
PATH=/media/nas/x21308/inst/texlive/2019/bin/x86_64-linux:/media/nas/x21308
/inst/texlive/2019/bin/x86_64-linux:/media/nas/x21308/inst/texlive/201
9/bin/x86_64-linux:/media/nas/x21324/miniconda3/bin:/media/nas/x21324/
miniconda3/condabin:/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/loc
al/bin:/usr/bin:/bin:/opt/bin:/usr/lib/mit/sbin:/opt/perforce/bin:/opt
/hdf-java/bin:/usr/lib/mit/sbin:/opt/perforce/bin
PYTHONSTARTUP=/home/gholl/.pythonrc.py
REQUESTS_CA_BUNDLE=<not set>
SOCKS_PROXY=<set>
SSL_CERT_FILE=<not set>
WINDOWPATH=7
XNLSPATH=/usr/share/X11/nls
ftp_proxy=<set>
gopher_proxy=<set>
http_proxy=<set>
https_proxy=<set>
no_proxy=<set>
socks_proxy=<set>
active environment : /media/nas/x21324/miniconda3
active env location : /media/nas/x21324/miniconda3
shell level : 1
user config file : /home/gholl/.condarc
populated config files : /home/gholl/.condarc
conda version : 4.8.2
conda-build version : not installed
python version : 3.8.1.final.0
virtual packages : __cuda=10.0
__glibc=2.26
base environment : /data/gholl/miniconda3 (writable)
channel URLs : https://conda.anaconda.org/rttools/linux-64
https://conda.anaconda.org/rttools/noarch
https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /data/gholl/miniconda3/pkgs
/home/gholl/.conda/pkgs
envs directories : /data/gholl/miniconda3/envs
/home/gholl/.conda/envs
platform : linux-64
user-agent : conda/4.8.2 requests/2.22.0 CPython/3.8.1 Linux/4.12.14-lp150.12.82-default opensuse-leap/15.0 glibc/2.26
UID:GID : 16209:16217
netrc file : None
offline mode : False
An unexpected error has occurred. Conda has prepared the above report.
If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers?
[y/N]:
Timeout reached. No report sent.
I expected a working conda environment with the latest released typhon and Python 3.8.1.
test_compress_decompress_zip
fails sometimes but not always.
from typhon.tests.files.test_utils import TestCompression
test = TestCompression()
passed = 0
for _ in range(50):
try:
test.test_compress_decompress_zip()
passed += 1
except Exception as error:
print(f"Passed {passed} times before failed:")
raise error
Passed 13 times before failed:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
typhon/typhon/tests/files/test_utils.py in test_compress_decompress_zip(self)
22 self.create_file(compressed_file)
23
---> 24 with decompress(file.name+".zip") as uncompressed_file:
25 assert self.check_file(uncompressed_file)
26
python36/lib/python3.6/contextlib.py in __enter__(self)
79 def __enter__(self):
80 try:
---> 81 return next(self.gen)
82 except StopIteration:
83 raise RuntimeError("generator didn't yield") from None
typhon/typhon/files/utils.py in decompress(filename, tmpdir)
179 try:
180 if fmt == 'zip':
--> 181 shutil.copyfileobj(compfile(filename, 'r').open(filebase, 'r'),
182 tmpfile,
183 chunksize)
python36/lib/python3.6/zipfile.py in open(self, name, mode, pwd, force_zip64)
1350 else:
1351 # Get info object for name
-> 1352 zinfo = self.getinfo(name)
1353
1354 if mode == 'w':
python3.6/zipfile.py in getinfo(self, name)
1279 if info is None:
1280 raise KeyError(
-> 1281 'There is no item named %r in the archive' % name)
1282
1283 return info
KeyError: "There is no item named 'tmpzu27cagz' in the archive"
Machine: platform linux -- Python 3.6.3, pytest-3.3.0, py-1.5.2, pluggy-0.6.0
typhon/typhon/arts/workspace/api.py
Lines 56 to 57 in c5eaa79
If the ARTS_INCLUDE_PATH
does not have a trailing delimiter, controlfiles are not found.
I will fix this ;) it is more about testing the issue mechanism
Cheers,
Lukas
The times returned by the NetCDF4
FileHandler
are wrong.
import os
import tempfile
import xarray as xr
import numpy as np
from typhon.files import NetCDF4
fh = NetCDF4()
with tempfile.TemporaryDirectory() as tdir:
tfile = os.path.join(tdir, "testfile.nc")
before = xr.Dataset(
{"a":
xr.DataArray(
np.array(
["2019-02-14T09:00:00", "2019-02-14T09:00:01"],
dtype="M8[ns]"))})
before["a"].encoding = {
"units": "seconds since 2019-02-14 09:00:00",
"scale_factor": 0.1}
fh.write(before, tfile)
after = fh.read(tfile)
assert np.array_equal(before["a"], after["a"])
The assertion fails.
The assertion should succeed.
The documentation is broken. First it demands a theme to be installed that is not installed when "pip install -e ." is run. Second it generates an error as below
Documentation behaves unexpectedly poorly
IN BASH IN MAIN TYPHON FOLDER:
cd doc
make html
pip install sphinx_rtd_theme
make html
# Paste your code here
#
#
Just a bunch of strange plots generated
# If applicable, paste the console output here
#
#
looking for now-outdated files...
Exception occurred:
File "/anaconda3/lib/python3.6/site-packages/sphinx/environment/init.py", line 782, in get_doctree/Dropbox/Software/typhon/doc/_build/doctrees/generated/DataFileError.doctree'
with open(doctree_filename, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '
The full traceback has been saved in /tmp/sphinx-err-zqk292vi.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at https://github.com/sphinx-doc/sphinx/issues. Thanks!
Makefile:55: recipe for target 'html' failed
make: *** [html] Error 2
NOTE: Also a lot of errors above here with functions or files missing
I want to have a .html file generated so I can see the documentation
Bug report for Freddy.
The workspace group ArrayOfVector
is not supported in the API, yet.
from typhon.arts.workspace import Workspace
ws = Workspace()
ws.ArrayOfVectorCreate('aov')
ws.VectorCreate('v')
ws.VectorSetConstant(ws.v, 2, 42.)
ws.Append(ws.aov, ws.v)
ws.aov.value
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
<ipython-input-6-452073056e06> in <module>()
6 ws.VectorSetConstant(ws.v, 2, 42.)
7 ws.Append(ws.aov, ws.v)
----> 8 ws.aov.value
~/Documents/typhon/typhon/arts/workspace/variables.py in value(self)
218 raise Exception("Variable of type " + self.group + " is empty.")
219 else:
--> 220 raise Exception("Type of workspace variable is not supported by the interface.")
221
222 def update(self):
Exception: Type of workspace variable is not supported by the interface.
It would be nice to support ArrayOfVector
as it is used for ybatch
results.
The NetCDF4
FileHandler
throws a ValueError
when trying to read masked dimensionless values. Those occur in GOES GLM L2 NetCDF files.
import xarray
from typhon.files import NetCDF4
ds = xarray.Dataset(
{"a":
xarray.DataArray(
42,
encoding={"_FillValue": 42})})
ds.to_netcdf("/tmp/spam.nc")
fh = NetCDF4()
ds2 = fh.read("/tmp/spam.nc")
Traceback (most recent call last):
File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/variable.py", line 79, in as_variable
obj = Variable(*obj)
File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/variable.py", line 259, in __init__
self._data = as_compatible_data(data, fastpath=fastpath)
File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/variable.py", line 176, in as_compatible_data
data[mask] = fill_value
ValueError: assignment destination is read-only
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "mwe.py", line 10, in <module>
ds2 = fh.read("/tmp/spam.nc")
File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/typhon/files/handlers/common.py", line 111, in wrapper
return method(*args, **kwargs)
File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/typhon/files/handlers/common.py", line 660, in read
self._load_group(dataset, None, root, fields)
File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/typhon/files/handlers/common.py", line 718, in _load_group
ds[path + var_name] = dims, var[:], dict(var.__dict__)
File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/dataset.py", line 980, in __setitem__
self.update({key: value})
File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/dataset.py", line 2505, in update
variables, coord_names, dims = dataset_update_method(self, other)
File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/merge.py", line 585, in dataset_update_method
indexes=dataset.indexes)
File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/merge.py", line 440, in merge_core
expanded = expand_variable_dicts(aligned)
File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/merge.py", line 214, in expand_variable_dicts
var = as_variable(var, name=name)
File "/hpc/uhome/gholl/miniconda3/envs/py37a/lib/python3.7/site-packages/xarray/core/variable.py", line 84, in as_variable
'{} to Variable.'.format(obj))
ValueError: Could not convert tuple of form (dims, data[, attrs, encoding]): ([], masked, {'_FillValue': 42}) to Variable.
The expected outcome would be to get an xarray.Dataset
object with the contents of the file.
SUSE Linux Enterprise Server 11 (x86_64), patchlevel 4
0.7.0+dev13.g0c85031
(latest git master at the time of writing)3.7.2
I installed most through conda, but llvmlite
, numba
, and typhon
through pip.
typhon/typhon/arts/workspace/methods.py
Lines 231 to 243 in c5eaa79
If a supergeneric method has implementations for more than two combinations of input/output, the current implementation fails to find anything but the first two combinations. E.g. the following implementations for Append exist:
Trying to call the 3rd form of Append currently fails with the error that such an overload cannot be found despite it existing.
Realised through FCDR_HIRS but the problem lies in typhon. On numpy 1.13:
$ plot_hirs_field_timeseries --plot_noise_with_other --without_gain --without_rself --temp_fields --count_fields space --with_corr above --corr_count 6 --corr_timeres 30min --sat noaa15 --channel 1 --from_date 1999-01-01 --to_date 1999-01-2
WARNING 2017-12-18 11:00:34,532 registry._define_single_adder:324: Redefining 'micro' (<class
'pint.definitions.PrefixDefinition'>)
WARNING 2017-12-18 11:00:34,532 registry._define_single_adder:324: Redefining 'ยต' (<class 'pin
t.definitions.PrefixDefinition'>)
INFO 2017-12-18 11:00:36,346 dataset.read_period:373: Reading fcdr_hirs {'satname': 'noaa15
'} for period 1999-01-01 00:00:00 โ 1999-01-02 00:00:00
[ ] N/A% (ETA: --:--:--) /
dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/numpy/ma/core.py:3008: FutureWarning
: Numpy has detected that you (may be) writing to an array returned
by numpy.diagonal or by selecting multiple fields in a structured
array. This code will likely break in a future numpy release --
see numpy.diagonal or arrays.indexing reference docs for details.
The quick fix is to make an explicit copy (e.g., do
arr.diagonal().copy() or arr[['f0','f1']].copy()).
if (obj.__array_interface__["data"][0]
/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/numpy/ma/core.py:3009: FutureWarnin
g: Numpy has detected that you (may be) writing to an array returned
by numpy.diagonal or by selecting multiple fields in a structured
array. This code will likely break in a future numpy release --
see numpy.diagonal or arrays.indexing reference docs for details.
The quick fix is to make an explicit copy (e.g., do
arr.diagonal().copy() or arr[['f0','f1']].copy()).
!= self.__array_interface__["data"][0]):
[=================== ] 27% (ETA: 0:00:11) W
ARNING 2017-12-18 11:00:41,122 filters.filter:151: 1 scanlines are out of order, resorting
WARNING 2017-12-18 11:00:41,132 filters.filter:159: 24 duplicate scanlines (judging from scanl
ine number), removing
[======================================================================] 100% (Time: 0:00:13)
Continues happily beyond this point. But on numpy 1.14, it fails at this point with:
Traceback (most recent call last):
File "/dev/shm/gerrit/venv/stable-3.6/bin/plot_hirs_field_timeseries", line 11, in <module>
load_entry_point('FCDR-HIRS==0.1.15+dev23.g827117e', 'console_scripts', 'plot_hirs_field_ti
meseries')()
File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/FCDR_HIRS/analysis/timeseri
es.py", line 1370, in main
temp_fields=p.temp_fields)
File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/FCDR_HIRS/analysis/timeseri
es.py", line 337, in __init__
NO_CACHE=True, **hrsargs)
File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/utils/cache.py", lin
e 54, in wrapper
return user_function(*args, **kwds)
File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/datasets/dataset.py"
, line 477, in read_period
arr = of.finalise(arr)
File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/datasets/filters.py"
, line 509, in finalise
fields_notclose = {nm for nm in rep.dtype.names
File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/datasets/filters.py"
, line 513, in <setcomp>
else numpy.isclose(rep[nm][0, ...], rep[nm])
File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/numpy/core/numeric.py", lin
e 2329, in isclose
dt = multiarray.result_type(y, 1.)
TypeError: invalid type promotion
There may be other places where typhon functionality fails on numpy 1.14. I suspect this is due to me doing something inappropriate to which numpy has become less tolerant, but this remains to be seen.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.