earthobservations / gribmagic Goto Github PK

Hi again,

while looking at #2, we found that the folders hosting parameters for temperature [1,2] and wind [3,4] not only contain model-level data, but also pressure-level data. So, we will probably have to improve model_variables_levels_mapping.yml and the respective downstream infrastructure to accept a list of labels on the right hand side of the mapping.

With kind regards,
Andreas.

[1] https://opendata.dwd.de/weather/nwp/icon/grib/00/t/
[2] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/t/
[3] https://opendata.dwd.de/weather/nwp/icon/grib/00/u/
[4] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/u/

Hi there,

within model_variables_levels_mapping.yml, temperature is designated as a model-level field. However, the model level is missing from the filename, so GribMagic croaks like

WARNING: Access failed: 404 Client Error: Not Found for url: [...]

The computed URL looks like [1] but it should be [2] instead.

With kind regards,
Andreas.

[1] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/t/icon-eu_europe_regular-lat-lon_model-level_2020121900_000_T.grib2.bz2
[2] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/t/icon-eu_europe_regular-lat-lon_model-level_2020121900_000_10_T.grib2.bz2

@mhaberler suggested at #4 (comment):

Note Cosmo-D2 is on the way out and is currently being replaced by Icon-D2 which is sort-of preproduction status (but stable as far as I can judge).

Thanks. So, we should also add acquisition for ICON-D2 to the mix.

Hi again,

we just found that data of DWD COSMO-D2 within the "temperature 2m" folder [1] is published with multiple grid variants like regular-lat-lon vs. rotated-lat-lon.

Parameter folders for "temperature" [2] also bring in the levels-axis like model-level vs. pressure-level as outlined within #3, so this folder and friends contains four (4) different sets of data in total.

As @mhaberler pointed out at #3 (comment), the docker-dwd-open-data-downloader by @EduardRosert also accounts for that by discriminating between "grids": ["regular-lat-lon", "rotated-lat-lon"], see models.json.

With kind regards,
Andreas.

[1] https://opendata.dwd.de/weather/nwp/cosmo-d2/grib/00/t_2m/
[2] https://opendata.dwd.de/weather/nwp/cosmo-d2/grib/00/t/

Hi there,

within model_variables_mapping.yml, diffuse_horizontal_irradiance is mapping to DWD ICON's aswdiff_s parameter field. However, there [1-4] are only folders called aswdifd_s and aswdifu_s.

With kind regards,
Andreas.

[1] https://opendata.dwd.de/weather/nwp/icon/grib/00/
[2] https://opendata.dwd.de/weather/nwp/icon-d2/grib/00/
[3] https://opendata.dwd.de/weather/nwp/icon-eu/grib/00/
[4] https://opendata.dwd.de/weather/nwp/icon-eu-eps/grib/00/

Hi. When i run the file i keep getting invalid syntax when i run python setup.py install.

Dear Michael,

based on your improvements to the dwd-open-data-downloader by @EduardRosert and @bjoern-reetz you published at EduardRosert/docker-dwd-open-data-downloader#2 (thanks a stack to all of you!), I've just added a demo of a humble pipeline machinery [1] which might resonate with your proposal at earthobservations/wetterdienst#271 (comment).

The goal is to have as much details available as Python code as possible. So, one detail that was important to me was to have the "download/processing recipe" available as Python code in order to lay the groundwork of evolving this into different directions, see [2].

It is really only a demo, completely unpolished yet and it might not even work on Windows. It has to be invoked from the toplevel directory. A short synopsis about installation and operation would be:

# Acquire sources
git clone https://github.com/earthobservations/gribmagic
cd gribmagic

# Install ECMWF ecCodes, Climate Data Operators (CDO) and ECMWF Magics. YMMV. See also `docs/magics.rst`.
# See also https://confluence.ecmwf.int/display/ECC and https://code.mpimet.mpg.de/projects/cdo.

# Linux
apt-get install --yes libeccodes0 cdo libmagplus3v5

# macOS
brew install eccodes cdo
make magics-install
make magics-info

# Python virtualenv FTW
python3 -m venv .venv
source .venv/bin/activate

# Install Python package
pip install --editable=.[plotting]

# Install improved DWD Open Data Downloader
gribmagic install dwd-grib-downloader

# Invoke the machinery
./demo/acquire_and_bbox.sh

While still in its infancy, I believed it would be a good idea to share this with you and will be happy about any feedback.

With kind regards,
Andreas.

[1] https://github.com/earthobservations/gribmagic/blob/8d5ffdc/demo/pipeline.sh
[2] https://github.com/earthobservations/gribmagic/blob/8d5ffdc/demo/pipeline/recipe.py

cc @gutzbenj, @wetterfrosch, @mhaberler

Demo plot

-- .gribmagic-data/subgrid/bbox_46.0_47.5_14.5_16.8/png/icon-d2_germany_regular-lat-lon_pressure-level_2020122212_001_950_v.grib2.png