Code Monkey home page Code Monkey logo

rockhound's Introduction

NOTICE: Development of RockHound has stopped.

Our focus has shifted to curating a set of data that has open licenses and serves as good sample data for tutorials and documentation. We're gathering and packaging these datasets in https://github.com/fatiando-data and have created the Ensaio package to load them.


RockHound

rock hound : noun

  1. A specialist in geology¹
  2. An amateur rock and mineral collector¹
  3. A Python library to download and read common geophysical models and datasets²

¹ Merriam Webster dictionary | ² Not a real dictionary definition.

Documentation | Documentation (dev version) | Contact | Part of the Fatiando a Terra project

Latest version on PyPI

Latest version on conda-forge

TravisCI build status

Azure Pipelines build status

Test coverage status

Compatible Python versions.

Digital Object Identifier for the Zenodo archive

Disclaimer

🚨 This package is in early stages of design and implementation. 🚨

We welcome any feedback and ideas! Let us know by submitting issues on Github or send us a message on our Gitter chatroom.

About

RockHound is a Python library to download geophysical models and datasets (PREM, CRUST1.0, ETOPO1) and load them into Python data structures (pandas, numpy, xarray).

Many of these models use non-conventional file formats or can be tricky to find on the internet. RockHound knows how to download them if you don't already have them locally, read the file format, and return a nicely formatted data structure. Under the hood, it uses Pooch to manage the downloads.

Project goals

  • Download commonly used models and datasets.
  • Load data into pandas.DataFrame (tables) and xarray.Dataset (grids).
  • Only download if needed and check downloads for corruption.
  • Provide functions for visualizing complex models and datasets.

Contacting Us

Citing RockHound

This is research software made by scientists (see AUTHORS.md). Citations help us justify the effort that goes into building and maintaining this project. If you used RockHound for your research, please consider citing us.

See our CITATION.rst file to find out more.

Contributing

Code of conduct

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

Contributing Guidelines

Please read our Contributing Guide to see how you can help and give feedback.

Imposter syndrome disclaimer

We want your help. No, really.

There may be a little voice inside your head that is telling you that you're not ready to be an open source contributor; that your skills aren't nearly good enough to contribute. What could you possibly offer?

We assure you that the little voice in your head is wrong.

Being a contributor doesn't just mean writing code. Equally important contributions include: writing or proof-reading documentation, suggesting or implementing tests, or even giving feedback about the project (including giving feedback about the contribution process). If you're coming to the project with fresh eyes, you might see the errors and assumptions that seasoned contributors have glossed over. If you can write any code at all, you can contribute code to open source. We are constantly trying out new skills, making mistakes, and learning from those mistakes. That's how we all improve and we are happy to help others learn.

This disclaimer was adapted from the MetPy project.

License

This is free software: you can redistribute it and/or modify it under the terms of the BSD 3-clause License. A copy of this license is provided in LICENSE.txt.

Documentation for other versions

rockhound's People

Contributors

aguspesce avatar fatiando-bot avatar leouieda avatar rowanc1 avatar santisoler avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

rockhound's Issues

More 1D Earth Models from IRIS

Description of the desired feature

There are a bunch of Earth Models in the IRIS website besides PREM (see #10). We could have functions to download and load many of those, which would be interesting to plot and see their differences. See http://ds.iris.edu/ds/products/emc-referencemodels/

These functions wouldn't be too complicated and load the models into pandas.DataFrame, much like the fetch_prem function.

Integration with Intake

Intake is a general-purpose python package for accessing and cataloguing data. It is in use, amongst other places, by the Pangeo geo-atmospheric collaboration to reference several datasets (see a nice rendering of the nested catalogue structure at https://pangeo-data.github.io/pangeo-datastore/ ) and has integrations with scientific data collection descriptions such as STAC, THREDDS and CMIP.

It seems that we have some things in common! Particularly, Intake cares about careful description of datasets, a unified user API for inspecting and loading those data (into memory or dask, various containers), caching for load-on-first-access.

Lets start a conversation here about possible a integration, e.g., that Intake would show the rockhound collections as one of its catalogues.

cc @rabernat

Slab2 - A Comprehensive Subduction Zone Geometry Model

I would like to request the Slab2 model to be added to Rockhound.
This model contains the three-dimensional geometries of all active global subduction zones.
This kind of data is important to create realistic simulations of subduction zones in geodynamics and correct gravity data.

Are you willing to help implement and maintain this feature? Yes

NOTICE: Development of RockHound has stopped

Hi everyone, I've been meaning to write this issue for a while but ended up forgetting. As you probably noticed, development of RockHound has pretty much stopped. The main reason is that the way we were going about adding datasets had several fatal flaws that made development unsustainable:

  • No scope definition. We basically didn't define a scope for the project and so had no way of indicating which datasets would fit. This is mostly because we didn't know and thought we could just add everything here with no problem.
  • CI and gallery infrastructure doesn't scale. As we started adding more datasets, our set up of downloading everything on CI to test and build the documentation started breaking. This is clearly not possible to do on free CI and would only get worse as the project grew.
  • Lack of open licenses. Initially, the goal was to download and read into pandas/xarray all of the data and models that exits without clear licenses and in terrible custom formats (like CRUST1.0). This works fine for small files but as soon as the pre-processing gets more involved, this solution becomes difficult to maintain. The best way would be to repackage the datasets into a format that can be easily loaded with pandas/xarray but since these data have no license, we can't do that.

So development efforts slowly fizzled out because of these mounting problems that we had no good way of fixing. As such, we realized that development was not sustainable. A better approach to these problems is to build packages for specific datasets. That solves the CI/growth issue. So if anyone wants to pick that up, please feel free to use whatever code we have that helps you do that 🙂

For the future, we decided to shift our focus to curating a set of data that has open licenses and serves as good sample data for tutorials and documentation. Our packages have this already but we wanted to consolidate on a single package that can be shared. This effort has a much tighter scope so it's easier to know if we're going in the right direction.

We're gathering and packaging these datasets in https://github.com/fatiando-data and have created the Ensaio package to load them.

Sorry for the delay in this message and thank you to everyone who contributed to RockHound! We will be closing most PRs and Issues and putting a notice on the README linking to this issue.

ETOPO1

Description of the desired feature

Include ETOPO1 from https://www.ngdc.noaa.gov/mgg/global/global.html

Load as xarray grids with option to choose the version between ice surface and bedrock. Quite a large download (~350Mb) so it might take a while the first time it's downloaded.

Change Zenodo url on MAINTENANCE.md

Description of the desired feature

Wouldn't be better to change the Zenodo url to the Fatiando a Terra community?
It would make clearer that the release should be done through the New upload button on the community.

Are you willing to help implement and maintain this feature? Yes

0.1.0 release

We're almost ready for an initial 0.1.0 release. The only thing left is to use the new pooch.Decompress processor instead of the custom one implemented here. This will require bumping the pooch requirement to >=0.5.0. After that, we need:

Stop versioning the data cache directory

Description of the desired feature

Maybe we shouldn't be versioning the data cache location. This triggers new downloads of all datasets every time we get a new release. Since these datasets aren't likely to change without releasing a new version, we probably don't need this.

Add Manville well log dataset

Description of the desired feature
Suggest adding the Mannville Group dataset of 2000+ well logs, tops, and facies labels from Alberta Geological Survey. It also has metadata and a report.
This dataset has been used in multiple hackathons. https://ags.aer.ca/publications/SPE_006.html

Different people with different goals might want to load different parts of that dataset. Here are some examples:

(1) Be able to load all data (excluding metadata and report) into a pandas dataframe for whatever purposes people might have
(2) Be able to load data that serves as a starter dataset for predicting tops. Different people might have different required curves or different tops, but there should be a starter dataset for that class of problem. For example, Top McMurray as the top and required curves are ['ILD', 'NPHI', 'GR', 'DPHI', 'DEPT'].
3) Be able to load all wells with facies + specific well log curves into a pandas dataframe for facies prediction. Required well log curves might be ['ILD', 'NPHI', 'GR', 'DPHI', 'DEPT'].

Are you willing to help implement and maintain this feature?

Yes.
Some useful starting points:

I'd like to make the data loading for Predictatops integrated with Pooch & Rockhound.

Avoid CIs from downloading large amount of data

Description of the desired feature

CIs now download all files in registry while running tests and building the gallery.

After new datasets will be included into Rockhound, the amount of data needed to be fetched may grow too large that CIs may be not be capable of finishing the testing builds.
Because there's no way to permanently store this datasets on the CIs, we decided to remove test and gallery builds from being run on by the CIs. Instead, the maintainers should run the tests and build the gallery locally before each PR is merged into master. Test functions and gallery examples must still be included on every new PR (and 100% coverage is still a goal to achieve), but the difference is that CIs won't run them.
All files inside doc/gallery should be pushed to the repository so CIs can still build the docs (and deploy them) but avoiding to rerun the gallery examples.
A new minimal test function should be added that only checks if all files in the registry are available for download.
Add an extra point to PR checklist to remind contributors to build the gallery example and push any change on doc/gallery, and maintainers to run all tests before merging the PR.

Are you willing to help implement and maintain this feature? Yes

Replace numpydoc for napoleon to unpin Sphinx

Description of the desired feature

The combination of latest sphinx (2.2.0), numpydoc, and sphinx_rtd_theme
causes a mangled rendering of the parameter list. We were pinning sphinx
to 1.8.5, which is getting old and causing dependency problems.
See discussion in fatiando/pooch#113.

A possible solution is to replace numpydoc with sphinx.ext.napoleon.
It solves the problem although it generates a slightly different output.
The class autosummary template must be modified in order to account for
missing methods summary block.

This replacement has been achieved by @leouieda on fatiando/pooch#122.
See the changes applied on that PR for inspiration.

ImportError : fetch_bedmap2 does not work with 1.0.25 of rasterio

When running the test suite the test for bedmap2 fails with the current version of rasterio. The issue appears to be in line 102 of fetch_bedmap2.py. A call to xr.open_rasterio requires an import from rasterio._base import gdal_version which fails. This problem can be resolved by using version 1.0.24 of rasterio.

Full code that generated the error

def test_bedmap2_multiple_datasets():
    "Load multiple datasets"
    # Test with two datasets with same shape
    grid = fetch_bedmap2(["bed", "surface"])
    assert set(grid.data_vars) == set(["bed", "surface"])
    # Test with "bed" and "lakemask_vostok"
    grid = fetch_bedmap2(["bed", "lakemask_vostok"])
    assert set(grid.data_vars) == set(["bed", "lakemask_vostok"])
    # Test with "bed" and "thickness_uncertainty_5km"
    grid = fetch_bedmap2(["bed", "thickness_uncertainty_5km"])
    assert set(grid.data_vars) == set(["bed", "thickness_uncertainty_5km"])

Full error message

________________________ test_bedmap2_multiple_datasets ________________________

    def test_bedmap2_multiple_datasets():
        "Load multiple datasets"
        # Test with two datasets with same shape
>       grid = fetch_bedmap2(["bed", "surface"])

../rockhound/tests/test_bedmap2.py:37: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../rockhound/bedmap2.py:102: in fetch_bedmap2
    array = xr.open_rasterio(get_fname(dataset, fnames), chunks=chunks, **kwargs)
/home/chet/anaconda3/envs/rockhound/lib/python3.7/site-packages/xarray/backends/rasterio_.py:214: in open_rasterio
    import rasterio
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """Rasterio"""
    
    from __future__ import absolute_import
    
    from collections import namedtuple
    from contextlib import contextmanager
    import logging
    
    try:
        from pathlib import Path
    except ImportError:  # pragma: no cover
        class Path:
            pass
    
    try:
        from logging import NullHandler
    except ImportError:  # pragma: no cover
        class NullHandler(logging.Handler):
            def emit(self, record):
                pass
    
>   from rasterio._base import gdal_version
E   ImportError: libtiledb.so.1.6.0: cannot open shared object file: No such file or directory

/home/chet/anaconda3/envs/rockhound/lib/python3.7/site-packages/rasterio/__init__.py:22: ImportError

System information

  • Operating system: Ubuntu 18.04.3 LTS
  • Python installation (Anaconda, system, ETS): Anaconda
  • Version of Python: 3.7.3
  • Version of this package: 0.10 (Failure on fetch_models WIP pull request)
  • If using conda, paste the output of conda list below:
output of conda list
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
affine                    2.2.2                      py_0    conda-forge
alabaster                 0.7.12                     py_0    conda-forge
appdirs                   1.4.3                      py_1    conda-forge
asn1crypto                0.24.0                py37_1003    conda-forge
astroid                   2.2.5                    py37_0    conda-forge
atomicwrites              1.3.0                      py_0    conda-forge
attrs                     19.1.0                     py_0    conda-forge
babel                     2.7.0                      py_0    conda-forge
black                     19.3b0                     py_0    conda-forge
bokeh                     1.3.4                    py37_0    conda-forge
boost-cpp                 1.70.0               h8e57a91_2    conda-forge
bzip2                     1.0.8                h516909a_0    conda-forge
ca-certificates           2019.6.16            hecc5488_0    conda-forge
cairo                     1.16.0            hfb77d84_1002    conda-forge
cartopy                   0.17.0          py37he1be148_1005    conda-forge
certifi                   2019.6.16                py37_1    conda-forge
cffi                      1.12.3           py37h8022711_0    conda-forge
cfitsio                   3.470                hb60a0a2_2    conda-forge
chardet                   3.0.4                 py37_1003    conda-forge
click                     7.0                        py_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.5.0                      py_0    conda-forge
cloudpickle               1.2.1                      py_0    conda-forge
cmocean                   2.0                        py_1    conda-forge
colorspacious             1.1.2              pyh24bf2e0_0    conda-forge
coverage                  4.5.4            py37h516909a_0    conda-forge
cryptography              2.7              py37h72c5cf5_0    conda-forge
curl                      7.65.3               hf8cf82a_0    conda-forge
cycler                    0.10.0                     py_1    conda-forge
cytoolz                   0.10.0           py37h516909a_0    conda-forge
dask                      2.3.0                      py_0    conda-forge
dask-core                 2.3.0                      py_0    conda-forge
dbus                      1.13.6               he372182_0    conda-forge
distributed               2.3.0                      py_0    conda-forge
docutils                  0.15.2                   py37_0    conda-forge
entrypoints               0.3                   py37_1000    conda-forge
expat                     2.2.5             he1b5a44_1003    conda-forge
flake8                    3.7.8                    py37_1    conda-forge
fontconfig                2.13.1            h86ecdb6_1001    conda-forge
freetype                  2.10.0               he983fc9_1    conda-forge
freexl                    1.0.5             h14c3975_1002    conda-forge
fsspec                    0.4.1                      py_0    conda-forge
geos                      3.7.2                he1b5a44_1    conda-forge
geotiff                   1.5.1                h560c3f3_2    conda-forge
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
giflib                    5.1.7                h516909a_1    conda-forge
glib                      2.58.3            h6f030ca_1002    conda-forge
gst-plugins-base          1.14.5               h0935bb2_0    conda-forge
gstreamer                 1.14.5               h36ae1b5_0    conda-forge
hdf4                      4.2.13            h9a582f1_1002    conda-forge
hdf5                      1.10.5          nompi_h3c11f04_1100    conda-forge
heapdict                  1.0.0                 py37_1000    conda-forge
icu                       64.2                 he1b5a44_0    conda-forge
idna                      2.8                   py37_1000    conda-forge
imagesize                 1.1.0                      py_0    conda-forge
importlib_metadata        0.18                     py37_0    conda-forge
isort                     4.3.21                   py37_0    conda-forge
jinja2                    2.10.1                     py_0    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
json-c                    0.13.1            h14c3975_1001    conda-forge
kealib                    1.4.10            h58c409b_1005    conda-forge
kiwisolver                1.1.0            py37hc9558a2_0    conda-forge
krb5                      1.16.3            h05b26f9_1001    conda-forge
lazy-object-proxy         1.4.1            py37h516909a_0    conda-forge
libblas                   3.8.0               12_openblas    conda-forge
libcblas                  3.8.0               12_openblas    conda-forge
libcurl                   7.65.3               hda55be3_0    conda-forge
libdap4                   3.20.2            hd48c02d_1000    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0  
libgdal                   3.0.1                hbef8c27_6    conda-forge
libgfortran-ng            7.3.0                hdf63c60_0  
libiconv                  1.15              h516909a_1005    conda-forge
libkml                    1.3.0             h4fcabce_1010    conda-forge
liblapack                 3.8.0               12_openblas    conda-forge
libnetcdf                 4.6.2             h056eaf5_1002    conda-forge
libopenblas               0.3.7                h6e990d7_1    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libpq                     11.5                 hd9ab2ff_0    conda-forge
libspatialite             4.3.0a            he1bb1e1_1029    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0  
libtiff                   4.0.10            h57b8799_1003    conda-forge
libuuid                   2.32.1            h14c3975_1000    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxml2                   2.9.9                hee79883_2    conda-forge
locket                    0.2.0                      py_2    conda-forge
lz4-c                     1.8.3             he1b5a44_1001    conda-forge
markupsafe                1.1.1            py37h14c3975_0    conda-forge
matplotlib                3.1.1                    py37_1    conda-forge
matplotlib-base           3.1.1            py37he7580a8_1    conda-forge
mccabe                    0.6.1                      py_1    conda-forge
more-itertools            7.2.0                      py_0    conda-forge
msgpack-python            0.6.1            py37h6bb024c_0    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
numpy                     1.17.0           py37h95a1406_0    conda-forge
numpydoc                  0.9.1                      py_0    conda-forge
olefile                   0.46                       py_0    conda-forge
openjpeg                  2.3.1                h58a6597_0    conda-forge
openssl                   1.1.1c               h516909a_0    conda-forge
owslib                    0.18.0                     py_0    conda-forge
packaging                 19.0                       py_0    conda-forge
pandas                    0.25.0           py37hb3f55d8_0    conda-forge
partd                     1.0.0                      py_0    conda-forge
pcre                      8.41              hf484d3e_1003    conda-forge
pillow                    6.1.0            py37h6b7be26_1    conda-forge
pip                       19.2.2                   py37_0    conda-forge
pixman                    0.38.0            h516909a_1003    conda-forge
pluggy                    0.12.0                     py_0    conda-forge
pooch                     0.5.2                    py37_0    conda-forge
poppler                   0.67.0               ha967d66_7    conda-forge
poppler-data              0.4.9                         1    conda-forge
postgresql                11.5                 hc63931a_0    conda-forge
proj4                     6.1.0                he751ad9_2    conda-forge
psutil                    5.6.3            py37h516909a_0    conda-forge
pthread-stubs             0.4               h14c3975_1001    conda-forge
py                        1.8.0                      py_0    conda-forge
pycodestyle               2.5.0                      py_0    conda-forge
pycparser                 2.19                     py37_1    conda-forge
pyepsg                    0.4.0                      py_0    conda-forge
pyflakes                  2.1.1                      py_0    conda-forge
pygments                  2.4.2                      py_0    conda-forge
pykdtree                  1.3.1           py37h3010b51_1002    conda-forge
pylint                    2.3.1                    py37_0    conda-forge
pyopenssl                 19.0.0                   py37_0    conda-forge
pyparsing                 2.4.2                      py_0    conda-forge
pyproj                    2.2.1            py37hc44880f_0    conda-forge
pyqt                      5.9.2            py37hcca6a23_2    conda-forge
pyshp                     2.1.0                      py_0    conda-forge
pysocks                   1.7.0                    py37_0    conda-forge
pytest                    5.1.0                    py37_0    conda-forge
pytest-cov                2.7.1                      py_0    conda-forge
python                    3.7.3                h33d41f4_1    conda-forge
python-dateutil           2.8.0                      py_0    conda-forge
pytz                      2019.2                     py_0    conda-forge
pyyaml                    5.1.2            py37h516909a_0    conda-forge
qt                        5.9.7                h0c104cb_3    conda-forge
rasterio                  1.0.25           py37h900e953_2    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
requests                  2.22.0                   py37_1    conda-forge
rockhound                 0.1.0+20.gddc8b2d           dev_0    
scipy                     1.3.1            py37h921218d_2    conda-forge
setuptools                41.0.1                   py37_0    conda-forge
shapely                   1.6.4           py37hec07ddf_1006    conda-forge
sip                       4.19.8          py37hf484d3e_1000    conda-forge
six                       1.12.0                py37_1000    conda-forge
snowballstemmer           1.9.0                      py_0    conda-forge
snuggs                    1.4.6                      py_0    conda-forge
sortedcontainers          2.1.0                      py_0    conda-forge
sphinx                    1.8.5                    py37_0    conda-forge
sphinx-gallery            0.4.0                    py37_0    conda-forge
sphinx_rtd_theme          0.4.3                      py_0    conda-forge
sphinxcontrib-websupport  1.1.2                      py_0    conda-forge
sqlite                    3.29.0               hcee41ef_0    conda-forge
tbb                       2018.0.5             h2d50403_0    conda-forge
tblib                     1.4.0                      py_0    conda-forge
tiledb                    1.6.2                h69c774e_0    conda-forge
tk                        8.6.9             hed695b0_1002    conda-forge
toml                      0.10.0                     py_0    conda-forge
toolz                     0.10.0                     py_0    conda-forge
tornado                   6.0.3            py37h516909a_0    conda-forge
tzcode                    2019a             h516909a_1002    conda-forge
urllib3                   1.25.3                   py37_0    conda-forge
viscm                     0.7                pyh24bf2e0_0    conda-forge
wcwidth                   0.1.7                      py_1    conda-forge
wheel                     0.33.4                   py37_0    conda-forge
wrapt                     1.11.2           py37h516909a_0    conda-forge
xarray                    0.12.3                     py_0    conda-forge
xerces-c                  3.2.2             h8412b87_1004    conda-forge
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libice               1.0.10               h516909a_0    conda-forge
xorg-libsm                1.2.3             h84519dc_1000    conda-forge
xorg-libx11               1.6.8                h516909a_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h14c3975_1007    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
yaml                      0.1.7             h14c3975_1001    conda-forge
zict                      1.0.0                      py_0    conda-forge
zipp                      0.5.2                      py_0    conda-forge
zlib                      1.2.11            h516909a_1005    conda-forge
zstd                      1.4.0                h3b9ef0a_0    conda-forge

Reduce memory consumption when building Bedmap2 example

Description of the desired feature

Generating the bedmap2 example when building the docs can consume a lot of memory an may crash not so powerful systems.
Would be nice to prevent this from happening, or at least raise a warning before start building the docs in order to avoid annoying situations.
On #45 we have reduced the memory consumption when reading the Bedmap2 datasets by converting them into Dask arrays, but generating the plot is still loading the entire file into RAM.

One solution proposed by @leouieda on #45 would be to plot just a portion of the continent.
Another one would be to reduce the resolution, although we must filter it to prevent aliasing.
Any new idea is very welcome!

Are you willing to help implement and maintain this feature? Yes

Drop support for Python 3.5

Conda-forge is no longer building 3.5 packages since October 2018. We should probably remove it from our CIs and officially drop support for new releases. This will mean updating:

  • doc/install.rst
  • setup.py
  • .travis.yml

Seafloor age grid

Description of the desired feature

The classic seafloor age grid by Müller et al: https://www.ngdc.noaa.gov/mgg/ocean_age/ocean_age_2008.html

There are many flavors (age, uncertainty, etc) but they're all in netCDF files. Loading them with xarray should be straight forward. Depending on their side, they can be downloaded individually (like in ETOPO1) if they're big or all at once if they are small (and loaded into a single Dataset).

[DATASET] SEG/EAGE Salt Dome Model

Desired dataset/model:

Are you willing to help implement and maintain this feature? Somewhat... @prisae, would you want to help me and we could both do this?


NOTE: The SEG website is having some issues and has been down for a few days, but the direct download link still works.

Also, here is a notebook we put together to show how to download and visualize the model and the various accompanying surface meshes using PyVista: https://nbviewer.jupyter.org/github/pyvista/show-room/blob/master/visualize-data.ipynb

(that example notebook might move around so let me know if the link breaks in the future)

Screen Shot 2019-07-01 at 12 08 20 AM

Screen Shot 2019-07-01 at 12 07 39 AM

LITHO1.0 model

I would like to request the LITHO1.0 model to be added to Rockhound.
This model is a 1° tessellated model of the crust and uppermost mantle of the earth, extending into the upper mantle to include the lithospheric lid and underlying asthenosphere. The model is parameterized laterally by tessellated nodes and vertically as a series of geophysically identified layers, such as water, ice, sediments, crystalline crust, lithospheric lid, and asthenosphere.

Model/dataset name: LITHO1 (https://igppweb.ucsd.edu/~gabi/litho1.0.html)
Download link: igppweb.ucsd.edu/~gabi/litho1/litho1.0.tar.gz
License: I dont know it
Target audience: geophysicists, solid earth, etc
Is it widely used: yes

Are you willing to help implement and maintain this feature? Yes

Dealing with Bedmap2 data type

Description of the desired feature

The Bedmap2 datasets are originally given in different data types. For example, the surface or bed datasets are int64, while the geoid is a float64.
Most of them include no data values, flagged as a great number.
When loading them, fetch_bedmap2() converts these no data values into nans.
This converts the data type of the dataset into float64 independently of its original data type.

Would be nice to at least have an optional argument in order to prevent the no data vals to be converted. This lets the user to get the datasets on the original data type.

Are you willing to help implement and maintain this feature? Yes

Decrease memory consumption when reading Bedmap2 dataset

Description of the desired feature

When reading the Bedmap2 dataset a lot of memory is consumed because the entire file is loaded into memory. This may cause system crashes and/or very annoying situations.

One way to overcome this situation is using Dask to read larger-than-memory data dividing it into chunks.

@leouieda I think you had more solutions in mind. Would you like to discuss them?

Are you willing to help implement and maintain this feature? Yes

Optimize file access and out-of-core reading

A lot of time and memory are wasted reading in whole large grids, like ETOPO1, even when only a small fraction is desired. The problem is that we store the compressed archives and need to unzip them in temporary folder every time. Since the files are temporary, we can't use xarray's lazy loading on the grids and must read them all into memory. This is not ideal.

This is a limitation of Pooch and need to think of a way of dealing with this. Ideas are welcome.

cc @santisoler

Support for common libraries' types

Description of the desired feature

With more 3D datasets to come like #50, it might be useful to have utilities in here that would make discretize, PyVista, PyGIMLi or GemPy meshes/types of these datasets.

While xarray is awesome, most users will want a type that they are familiar with and having a clear way to use these datasets with common libraries like those mentioned would help gain users (and developers).

Are you willing to help implement and maintain this feature? Yes/No

Definitely! But I'll need guidance on what these methods should be called and where/how they should be implemented.

Perhaps some sort of to_pyvista(), to_discretize(), etc. methods that will only execute if the needed library is available - thus keeping `rockhounds dependencies to a minimum?

Add license and citation on gallery examples and/or docstring

Description of the desired feature

Would be nice if the gallery example corresponding to each dataset would explicit the license under it's released and a reference to its article (or doi). By facilitating this information we could encourage people to cite the models they use, and get to know under which license are they released.

Are you willing to help implement and maintain this feature? Yes

Wrap all docstrings to 79 characters

Description of the desired feature

After fatiando/verde#177 raised by @prisae, fatiando/community#9 and fatiando/community#10 it has been decided that all docstrings must be wrapped to 79 characters per line.
We should warp all the existing docstrings in Rockhound to 79 characters and also configure flake8 to check for this when running the style checks.

Sadly, there's no way (yet) to automatically change all docstrings, but at least we can use flake8 to raise lines that fail. This can be done by setting max-doc-length to 79 characters on the flake8 configuration under setup.cfg.

Setup release drafter

In the last community call we decided to roll out release drafter on RockHound to test the automated changelog generation (see fatiando/contributing#15). I setup the following labels for this repository:

labels

I would propose we configure all repos to have the same labels (with same colors) to make things easier for us. I'll open a PR to enable the Github Actions workflow for this so we can discuss other configuration options.

Are these labels OK?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.