I added an option in the pyx file shown:

https://github.com/snowman2/pygrib/commit/fe12cd8b7aac442b6f79e97ab5b593d77b4237ad

However, I am not familiar enough with the connections needed to be made in the c file to feel comfortable making changes yet, so that part will need to be done.

Anyway, there it is. Use it if it is useful.

I've noticed the National Weather Service NDFD grib files have a tag "stepRange" which seems to be causing the pygrib created variable "validDate" to be set incorrectly to the end of the "stepRange".

While not a huge issue, it did cause some confusion on the intended valid times of the messages. I'm not sure if this is a bug in pygrib, or maybe NWS is encoding their messages/files in some awkward way. It does look like the section of code that creates "validDate" assumes stepRange always indicates the proper validTime, which perhaps is not be a safe assumption?

I successfully installed pygrib2.0.2 by pip, but when I import it in python, I will met the problem:

>>> import pygrib

Traceback (most recent call last):
File "", line 1, in
ImportError: dlopen(/Users/xxx/anaconda/lib/python2.7/site-packages/pygrib.so, 2): Library not loaded: @rpath/libmfhdf.0.dylib
Referenced from: /Users/xxx/anaconda/lib/libnetcdf.11.dylib
Reason: image not found

I tried in python2.7.13, 3.6.1, and the way of installing it I tried pip, conda. But I will always meet this problem. So how to solve the problem?

I have a use case which relies on interpretation of those two values as seen in versions prior to version 1.9.8. Would it be possible to make this behavior configurable within pygrib so that the library could be directed to follow grib_api standard or follow the interpretation of those values? I don't know if this would take the form of an argument to open, an alternative open function, a internal pygrib variable or what would make the most sense. I do not think there is a similar precedent to follow for pygrib.

I have encountered an incompatibility issue between pygrib latlons() and NDFD oceanic grids (GRIB2 files). The longitudes do not seem to be read in correctly using latlons(). It seems that the result of the command is that longitudes range from 129.906005 to 10.71. The longitudes should range from 129.906005 East to 10.71 East, but contain either negative longitudes or longitudes greater than 180 to indicate degrees West.

In other words, the minimum and maximum longitudes using latlons() is 10.71 and 129.906005, respectively, when they really should range from either -180 to 180 or 0 to 360 depending on the convention used for longitude.

The below few lines provide an example to reproduce the error. You can download an example NDFD oceanic GRIB2 file here: http://tgftp.nws.noaa.gov/SL.us008001/ST.expr/DF.gr2/DC.ndfd/AR.oceanic/VP.001-003/

import pygrib
#GRIB2 file downloaded from http://tgftp.nws.noaa.gov/SL.us008001/ST.expr/DF.gr2/DC.ndfd/AR.oceanic/VP.001-003/, which is from https://para.ocean.weather.gov/opc_gridded_marine.php:
grib='ds.wspd.bin';
grbs=pygrib.open(grib)
spd10m = grbs.select(shortName='ws',typeOfLevel='surface',level=0)[0].values
lat,lon = grbs[1].latlons()
print lon[800][800]
print lat[800][800]
print lon.min(),lon.max(),lat.min(),lat.max()

The result of the above script outputs the following for me:

92.0058444277 40.7152444663 10.71 129.906005 -30.4192 80.01

For comparison, wgrib2 outputs the following:

[opc_test@vm-lnx7-opctest2: /common/data/products/NDFD/grib2/exp/oceanic_day1to3]> wgrib2 -d 1 -ijlat 800 800 ds.wspd.bin 1:80:(800,800),lon=206.377544,lat=40.642622,val=9.999e+20 [opc_test@vm-lnx7-opctest2: /common/data/products/NDFD/grib2/exp/oceanic_day1to3]>

So, longitude at the (800,800) grid point is 206.377544 using wgrib2 (longitudes range from 0 to 360), whereas pygrib's latlons() shows a longitude of 92.0058444277 at (800,800) (longitudes decrease/range from 129.906005 to 10.71).

Using wgrib2 to convert to netcdf (wgrib2 -netcdf) correctly outputs longitudes ranging from 0 to 360.

Initial response received is that running the utility 'grib_list' on one of those files, you'll get:

latitudeOfFirstGridPointInDegrees = -30.4192 longitudeOfFirstGridPointInDegrees = 129.906005 latitudeOfLastGridPointInDegrees = 80.01 longitudeOfLastGridPointInDegrees = 10.71

The potential problem is that the domain crosses the Greenwich meridian, and the code does not work correctly when the longitudes are not monotonically increasing.

It seems that either the metadata in the file could be incorrect, or the grib_api C library could be interpreting it incorrectly.

Thanks,
Greg

Page for Cygwin installation instructions is missing.

Hi
I use pygrib to plot charts from GFS grib2 files (a UK subset). This works great and quick enough for my purposes. I am now looking into plotting time series of point data from the same grib files, but am finding that the process of extracting the data values for a given point to be far to slow.

I was hoping to create time series plots of variables such as temperature and winds on the fly - but on my system it takes over 30 seconds to get and process the data for one grid point from the first timestep to the 81st (10 days).

I have tried looping through individual grib files for each time step, extracting the values for the desired grid point and storing in a list. I also tried concatenating the grib files into one master grb2 file that contains all time steps, then looping through the messages for a given variable, but this was much slower.

I am used to using GrADS where a full gribfile for all timesteps could be opened, the desired timesteps selected (set t 1 81) and then a plot produced for these times for a given variable. Is a similar usage possible using pygrib/matplotlib?

For example, is there a way to specify the lat/lon subset required at the open/index stage, or can this only be done to individual grib messages eg grb.data(lat1=lat,lat2=lat,lon1=lon,lon2=lon)? I searched through all the docs but can't see that you deal with time sequences for point data anywhere. I'd be grateful for any pointers you can provide.
cheers
Dan

There is no support for the grid definition template 12. I'm working on a patch and can submit when I've got it to work (wont be fully tested though).

Otherwise, thanks for making the python grib2 interface.

Cheers,
Faron

building 'pygrib' extension
gcc -fno-strict-aliasing -I/Users/user/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/user/anaconda/lib/python2.7/site-packages/numpy/core/include -Ig2clib_src -I/Users/user/anaconda/include/python2.7 -c pygrib.c -o build/temp.macosx-10.7-x86_64-2.7/pygrib.o
In file included from /Users/user/anaconda/lib/gcc/x86_64-apple-darwin11.4.2/4.8.5/include-fixed/syslimits.h:7:0,
from /Users/user/anaconda/lib/gcc/x86_64-apple-darwin11.4.2/4.8.5/include-fixed/limits.h:34,
from /Users/user/anaconda/include/python2.7/Python.h:19,
from pygrib.c:4:
/Users/user/anaconda/lib/gcc/x86_64-apple-darwin11.4.2/4.8.5/include-fixed/limits.h:168:61: fatal error: limits.h: No such file or directory
#include_next <limits.h> /* recurse down to the real one */
^
compilation terminated.
error: command 'gcc' failed with exit status 1

From [email protected] on March 11, 2010 00:44:29

What steps will reproduce the problem? Download any GRIB2 file from ftp://ftp.ssec.wisc.edu/pub/eosdb/dbcras/ ,
e.g. ftp://ftp.ssec.wisc.edu/pub/eosdb/dbcras/2010_03_08_067/08Mar10.12z.dbCRAS.SouthAfrica.grib2 == code snippet starts ==

grbs = pygrib.open('08Mar10.12z.dbCRAS.SouthAfrica.grib2')
for grb in grbs:
lats, lons = grb.latlons()

== code snippet ends ==

Error message:

File "pygrib.pyx", line 771, in pygrib.gribmessage.latlons (pygrib.c:7680)
File "pygrib.pyx", line 539, in pygrib.gribmessage.getitem
(pygrib.c:4358)
RuntimeError: Key/value not found

I am using the SVN trunk version of pygrib, compiled against ECMWF grib_api
version 1.8 on 32-bit Ubuntu 9.10.

ps. Thanks Jeff, for fixing the issues so promptly. Please provide any additional information below.

Original issue: http://code.google.com/p/pygrib/issues/detail?id=3

Hi
I am having trouble using files made from cfsr timeseries data. I have used wgrib 2 to make a yearly set of grb files averaging 2m temp over a 30yr period.

When I look at these with wgrib2 all seems ok:
1:0:vt=2010123119:2 m above ground:30@1 year ave(1 hour fcst),missing=0:TMP Temperature [K]:
ndata=154481:undef=641:mean=273.014:min=239.389:max=303.231
grid_template=0:winds(N/S):
lat-lon grid:(641 x 241) units 1e-06 input WE:SN output WE:SN res 48
lat 20.000000 to 80.000000 by 0.250000
lon 280.000000 to 80.000000 by 0.250000 #points=154481
:vt=20101231190000

but when i try and open with pygrib I get the following error:

for grb in g: grb
...
GRIB_API ERROR : unable to get endStep as long (Decoding invalid)
Traceback (most recent call last):
File "", line 1, in
File "pygrib.pyx", line 439, in pygrib.open.next (pygrib.c:3574)
File "pygrib.pyx", line 678, in pygrib._create_gribmessage (pygrib.c:7655)
File "pygrib.pyx", line 721, in pygrib.setdates (pygrib.c:8155)
File "pygrib.pyx", line 1270, in pygrib.gribmessage.getitem (pygrib.c:17013)
RuntimeError: Decoding invalid

Any ideas if there is anything I can do to these files to enable use with pygrib? I did get the same error with similar t850 files, but managed to fix this using wgrib2's set_metadata. Unfortunately this trick is not working with this 2m temperature data set. The only difference is that the 2m temp data set is a 1hr forecast, and the t850 data was analysis data.

Thanks for any help you can offer - I've attached an example grib file

test_new.zip

Dan

I am trying to download a noaa grib2 file in python2.7 on Ubuntu 16.04. I am getting the following error:

Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import pygrib
YYYYMMDD=20170710
CC=00
file='http://www.ftp.ncep.noaa.gov/data/nccf/com/etss/prod/etss.${YYYYMMDD}/etss.t${CC}z.stormtide.ala3km.grib2'
gr=pygrib.open(file)
Traceback (most recent call last):
File "", line 1, in
File "pygrib.pyx", line 394, in pygrib.open.cinit (pygrib.c:2948)
IOError: [Errno could not open %s] http://www.ftp.ncep.noaa.gov/data/nccf/com/etss/prod/etss.${YYYYMMDD}/etss.t${CC}z.stormtide.ala3km.grib2

I get the same error on python3.
Can anyone provide some feedback on this error message. I'm a beginner in using pygrib.
Thanks

Hi,
On my local I have pygrib 1.9.8 is installed and I have the script to process grib2 file format. It works perfectly on my local but when I try to run it on another machine with pygrib 2.0.0 installed, I am getting "grib_accessor_class_codetable.c at line 316: assertion failure Assert(isdigit(*p))" message and script doesn't proceed or exit. I get the message when I call
grib2.select(name='U component of wind')
What may cause this message?

I installed the pygrib (2.0.2) from pip, but it fails in the tests.
I tried from repository too, following all instructions, same error.

$python test.py ... Trying: datsubset,latsubset,lonsubset=grb.data(lat1=15,lat2=65,lon1=220,lon2=320) Expecting nothing *** Error inpython': free(): corrupted unsorted chunks: 0x00000000020ed580 ***
Aborted (core dumped)`

dbg output here:
http://pastebin.com/1fTS9hGM

It fails to read my data too.
Am I doing something wrong, any reason, advice?

Hi, I am having difficulties building setup.py file. It seems that I am having trouble with the Jasper package. Below are the exact commands I executed to this point. Perhaps I have missed some steps?

Installing GFortran Compiler

sudo apt-get update
sudo apt-get install gcc gfortran g++ build-essential

Installing Jasper Package

wget http://www.ece.uvic.ca/~frodo/jasper/software/jasper-1.900.1.zip
unzip jasper-1.900.1.zip
cd jasper-1.900.1
./configure
make
make install

Installing GRIB Dependencies and GRIB API Package

sudo apt-get install libgrib-api-dev
sudo apt-get build-dep libgrib-api-dev
sudo apt-get install libgrib-api-tools
sudo apt-get install python-numpy
sudo apt-get install python-pyproj
sudo apt-get install python2.7-dev

Building PyGrib Package

wget https://github.com/jswhit/pygrib/archive/master.zip
unzip master.zip
cd pygrib-master
cp setup.cfg.template setup.cfg
nano setup.cfg

The following was uncommented:
#grib_api_dir=/usr/local/
#jasper_dir=/usr/local/
#png_dir=/usr
#zlib_dir=/usr

python setup.py build

Whenever I attempt to import pygrib I get the error:
dlopen(/Users/usrname/anaconda/lib/python2.7/site-packages/pygrib.so, 2): Library not loaded: @rpath/./libgrib_api.1.dylib Referenced from: /Users/usrname/anaconda/lib/python2.7/site-packages/pygrib.so Reason: image not found

I was using it fine yesterday with zero issues. I'm on a Mac running OSX 10.10.5 and using Python 2.7.

Is there a way to install pygrib and pygrib2 together with Python3.6?
Attempts to install conda install -c conda-forge pygrib=2.0.1 results in:
UnsatisfiableError: The following specifications were found to be in conflict:

pygrib 2.0.1*
python 3.6*

would it be possible/difficult to support fromstring() entire grib file instead of a single message?

Just wanted to ask what the future plans of this package are in light of the ecCodes development and the proposed Python interface?

:~/python/pygrib-master# python setup.py build
ok
ok
running build
running build_py
creating build/lib.linux-x86_64-2.7
copying ncepgrib2.py -> build/lib.linux-x86_64-2.7
running build_ext
building 'pygrib' extension
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/aichang/anaconda/lib/python2.7/site-packages/numpy/core/include -Ig2clib_src -I/home/aichang/anaconda/include/python2.7 -c pygrib.c -o build/temp.linux-x86_64-2.7/pygrib.o
In file included from /home/aichang/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1804:0,
from /home/aichang/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
from /home/aichang/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from pygrib.c:313:
/home/aichang/anaconda/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by "
^
pygrib.c:314:22: fatal error: grib_api.h: 没有那个文件或目录
#include "grib_api.h"
^
compilation terminated.
error: command 'gcc' failed with exit status 1

Is it possible to convert a grib1 file to grib2 using pygrib?
This is not really an issue, just a question.

Thanks!

When the environment variables GRIBAPI_DIR, GRIBAPI_LIBDIR, or GRIBAPI_INCDIR are set, the relevant directories are not added to libdirs and incdirs. As a result, python setup.py build and python setup.py install fail.

This issue was observed on OSX El Capitan with python 2.7.9 and python 3.4.2.

Is there any documentation regarding the subject matter? I am trying to plot upper wind from GFS data. I have not found any documentation for this.

Hi jswhit,

I would like to modify the regional 'value' of one gribmessage in a GRIB file while keeping other 'value' unchanged, but the method which works in normal numpy environment (s=gribmessage.value[lat1:lat2,lon1:lon2] s[:,:]=10.) seems does not work with gribmessage.value, only changing the whole field all at once will make the it happened

So is there a simpler way to directly change the regional gribmessage.value ?

From jakob.malm on May 04, 2010 06:41:20

I would like to be able to search for specific fields. GRIB API has this
functionality through the grib_index object since 1.8.0. I would suggest
that a "find" function be added to the pygrib module directly which takes a
filename and then **kwargs:

import pygrib
selected_grbs = pygrib.find('grib_filename', paramId=130, levtype="pl",
level=100)
for grb in selected_gribs:
# do something

This is much the same way that the GRIB API is constructed.

Original issue: http://code.google.com/p/pygrib/issues/detail?id=5

I have grib messages that use two different grids (one regular global grid and one gaussian global grid, and different numbers of lats and lons). I'd like to interpolate values on one grid to the other grid, replace the old grid in the grib message, and update the coordinates in the grib message accordingly. What's the proper procedure for doing that, or is that even allowed? I haven't found anything in the documentation about it yet. I can update the values to be a differently-shaped array (not sure if that's a bug), but pygrib won't allow me to touch latitudes and longitudes directly. Here's what I've tried:

1. Updating the grid info (latitudeOfFirstGridPointInDegrees, longitudeOfFirstGridPointInDegrees, latitudeOfLastGridPointInDegrees, longitudeOfLastGridPointInDegrees, Ni, and Nj). This is the closest I've gotten, but the latitudes are still incorrect, I suspect because of the gridType. Changing gridType raises an exception in latlons() because of latitudes being a scalar when an array is expected.
2. Keeping around a "template" grib message that has the correct latitudes and longitudes and changing the values and all the other read/write keys in the template. This seg faults for some reason after processing several grib messages.

Any help is appreciated!

Hi,

Is there a way to select byte ranges in a grib2 file on a remote server if there is no inventory file? In other words, can an inventory of messages be built without downloading the entire grib2 file?

In theory, there must be a way to do one pass to read only message headers (and get the byte ranges), but not sure if pygrib provides a way to do that.

Thanks.

From [email protected] on May 07, 2010 02:18:12

I have a GRIB2 data file
( ftp://ftp.ssec.wisc.edu/pub/eosdb/dbcras/2010_05_06_126/06May10.12z.dbCRAS G.SouthAfrica.grib2) which contains grid data (140x210) in polar
stereographic projection.

It appears that pygrib latlons() method automatically reprojects the grid
point coordinates from stereographic to geographic (latitude/longitude).

The 'wgrib2' utility ( http://www.cpc.noaa.gov/products/wesley/wgrib2/ ) is
also capable of transforming the grid point coordinates.

But, some return values of latlons() differ from those in the CSV dump of
'wgrib2'.

For ease of comparison, I have put the coordinate values side-by-side in a
single text file (dbcras-lonlats-compare.txt) in the following format:

"lon-lat from wgrib2" "lon-lat from pygrib"

To illustrate, below is a snippet the file:
-4.50056,-45.7956 -4.514481,-45.789450
-4.02109,-46.0063 -4.035438,-46.000016
-3.53655,-46.2144 -3.551329,-46.208080
-3.04692,-46.4199 -3.062145,-46.413584
-2.55221,-46.6228 -2.567878,-46.616468
-2.0524,-46.8231 -2.068522,-46.816676
-1.54749,-47.0206 -1.564075,-47.014148
-1.03749,-47.2153 -1.054539,-47.208826
-0.522396,-47.4071 -0.539918,-47.400650
-0.00222074,-47.596 -0.020220,-47.589561

As you can see, some of them vary quite significantly.

I am a newbie in Python and GIS field. Do you know what could possibly
account for the variation?

I am using 32-bit Ubuntu 10.04 with wgrib2 v0.1.8.2 and pygrib SVN revision
192 .

Attachment: dbcras-lonlats-compare.txt

Original issue: http://code.google.com/p/pygrib/issues/detail?id=6

I just want to modify some data of one grib2 file and then write modified data and the others to a new grib2 file. Is there any way to achieve this goal?

I am able to extract the data for a particular region with the help of documentation. How do I extract for a specif location say lat 13.0 and lon 80.0?

I'm unable to run a successful test suite using Python 3.5. Has anyone else had better luck/success?

I'm attaching both my (test) conda environment and the output from running the test suite. I'm using grib-api build/installed via homebrew.

env.yaml.txt
test.log.txt

I have added a .txt extension to allow for uploading in Github.

Running into an issue here that I can't imagine is that uncommon given the large number of users in the meteorological community. I work with datasets between each of the .nc, .grib and .hdf4 formats and it seems an oversight that there is non-conformity between this dependency for the three - this wasn't an issue in pygrib 1.9.9 as far as I'm aware (used to have all 3 playing nice):

In [1]: import pygrib

ImportError Traceback (most recent call last)
in ()
----> 1 import pygrib

ImportError: libjpeg.so.9: cannot open shared object file: No such file or directory

Now - following a standard conda-forge install for pygrib you end up with jpeg9b which provides the above dependency issue. If you do this however, you get the same error for netCDF4 and pyhdf except for libjpeg.so.8. Not sure if there is a good resolution for this as I haven't been able to find out, but would prefer to be able to work between the formats as much as possible without relying on the dependency mess that is PyNIO.

I try this
...

data, lats, lons = grb.data(lat1=20,lat2=70,lon1=0,lon2=28)
grb['values'] = data
msg = grb.tostring() 
grb = pygrib.fromstring(msg)
data = grb.values()

Script dies with:

File "./get_latest.py", line 81, in open_grib_get_area
data= grb.values()
File "pygrib.pyx", line 850, in pygrib.gribmessage.getattr (pygrib.c:8490)
File "pygrib.pyx", line 1270, in pygrib.gribmessage.getitem (pygrib.c:14140)
File "pygrib.pyx", line 1390, in pygrib.gribmessage._reshape_mask (pygrib.c:15560)
ValueError: total size of new array must be unchanged

What else I should update in GRIB2, if I attempt writing a subset of the grid in a separate file? Sorry if this not a pygrib question.

Why did I meet "Import Error" with anaconda ipython notebook when I tried to import pygrib?

I have installed the pygrib via macport. (sudo port install py34-pygrib)

Thanks in advance.

Thanks for your work on pygrib! Here's a warning I sometimes see with current pygrib and numpy 1.12.0

/usr/lib64/python2.7/site-packages/ncepgrib2.py:738:
MaskedArrayFutureWarning: setting an item on a masked array which has a
shared mask will not copy the mask and also change the original mask array
in the future.
Check the NumPy 1.11 release notes for more information.
  fld[1::2,:] = fldsave[1::2,::-1]

I just installed pygrib. I get this error when importing it:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: libgrib_api.so: cannot open shared object file: No such file or directory

I have libgrib_api.so located in /usr/local/lib.
grib_api is located in /usr/local/share. This is my setup.cfg file:

[directories]
grib_api_dir = /usr/local/share
# install man pages for command line utilities here
man_dir = /usr/local/man
[files]
# uncomment and set to grib_api library name.
# If you are using GRIB-API with MSVS, library name might be 'grib_api_lib'.
grib_api_libname = grib_api_lib

I am not sure about the value I should be setting grib_api_libname to. Perhaps that's the issue.

This is a very common issue when installing several packages from a pip requirements file.

see scipy/scipy#453

Should be solvable by using a solution like https://github.com/pydata/pandas/blob/master/setup.py#L92.

This introduces a setuptools dependency if not handled carefully.

Hi,

I'm attempting to create a new grib file using an older one as a template and replacing the values in the messages with new ones. I've done this in the past and had it work, but am getting segmentation faults for the current data I'm working with. It looks to me like the data in the file I'm using as a template only has two significant digits, and my thought is that there isn't enough memory allocated in the file to hold the higher precision values I'm replacing into the message. Is there a known work around to this, or would you recommend creating the updated files in another way?

Thanks for any insight you can provide (and for creating pygrib!). The error message and commands leading to it are attached.

From [email protected] on April 13, 2010 07:42:56

What steps will reproduce the problem?:

In [3]: grbs = pygrib.open("gfs2010041300.grb")
In [4]: grb = grbs.message(1)
In [5]: grb = grbs.message(0)
Segmentation fault

I realize that the grib messages are not 0-indexed, however maybe a IndexError should be
thrown instead of segfaulting (or 0-index messages, but I guess that would be too backwards
incompatible). What version of the product are you using? On what operating system? 1.5, OS X 10.6 built with -arch x86_64

Original issue: http://code.google.com/p/pygrib/issues/detail?id=4

I'm trying to convert GRIB2 to GRIB1 using the cnvgrib2to1 located at the utils folder. I'm getting the data file from this link: ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/hourly/prod/ but it seems that it is not successful. I would appreciate any help with this!

I'm a member of the open-source Pillow imaging library for Python.

https://github.com/python-pillow/Pillow

I'm trying to increase test coverage of the 20-year-old-plus code base by adding unit tests for more file formats.

Pillow has very limited GRIB support: only a stub that can recognise the format. Read and write would require an extra handler. Nevertheless, I'd like to test what is there.

Would it be possible to use and redistribute GRIB files from https://github.com/jswhit/pygrib/tree/master/sampledata/ in the Pillow codebase as part of the test suite?

Pillow uses an open source PIL Software License: https://github.com/python-pillow/Pillow/blob/master/LICENSE

Thank you!

From [email protected] on February 16, 2010 09:15:58

After a standard installation of grib_api (on openSUSE 11.2 x86_64)

./configure --prefix=/opt/grib_api-1.8.0
make
make check
make install

trying to install pygrib-1.0 yields

python setup.py install
running build
running build_ext
building 'pygrib' extension
gcc -pthread -shared build/temp.linux-x86_64-2.6/pygrib.o
-L/opt/grib_api-1.8.0/lib -L/opt/grib_api-1.8.0/lib64 -L/usr/lib64
-lgrib_api -lpython2.6 -o build/lib.linux-x86_64-2.6/pygrib.so
/usr/lib64/gcc/x86_64-suse-linux/4.4/../../../../x86_64-suse-linux/bin/ld:
/opt/grib_api-1.8.0/lib/libgrib_api.a(grib_context.o): relocation
R_X86_64_32 against `a local symbol' can not be used when making a shared
object; recompile with -fPIC
/opt/grib_api-1.8.0/lib/libgrib_api.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

I tried to set CFLAGS="-fPIC" before installing grib_api -- then both the
installation of grib_api and pygrib work without any errors, but I get an
error when I import pygrib:

python
Python 2.6.2 ( r262 :71600, Oct 24 2009, 03:15:21)
[GCC 4.4.1 [gcc-4_4-branch revision 150839 ]] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import pygrib
Traceback (most recent call last):
File "", line 1, in
ImportError: /usr/local/lib64/python2.6/site-packages/pygrib.so: undefined
symbol: jas_matrix_create

probably because libjasper is not referenced in pygrib.so

ldd /usr/local/lib64/python2.6/site-packages/pygrib.so
linux-vdso.so.1 => (0x00007fff56de1000)
libpython2.6.so.1.0 => /usr/lib64/libpython2.6.so.1.0
(0x00007f7416b0a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f74168ed000)
libc.so.6 => /lib64/libc.so.6 (0x00007f7416592000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f741638e000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f741618a000)
libm.so.6 => /lib64/libm.so.6 (0x00007f7415f35000)
/lib64/ld-linux-x86-64.so.2 (0x00007f74171da000)

So how has grib_api to be compiled so that it is suitable for pygrib?

Original issue: http://code.google.com/p/pygrib/issues/detail?id=1

ECMWF has fully moved to the new ecCodes library now (which provides its own python interface), and will only provide limited support for GRIB_API, e.g. tables and urgent bug fixes, but this will end in the autumn of 2018.
What will happen with pygrib after that? Will it be discontinued? Or will you provide a wrapper to the new ecCodes library?

Hello,

I have recently discovered that pygrib 1.9.8 is unable to read GRIB2 files that the National Centers for Environmental Prediction (NCEP) provides for the operational HRRR model. These files are available at http://www.ftp.ncep.noaa.gov/data/nccf/nonoperational/com/hrrr/prod/. Here is some example code:

>>> import pygrib
>>> igrb = pygrib.open('hrrr.t00z.wrfsfcf05.grib2')
>>> rr = igrb.message(1)
>>> rr
1:Maximum/Composite radar reflectivity:dB (instant):lambert:unknown:level 0 10:fcst time 5 hrs:from
201511050000
>>> rr_array = rr.values
Segmentation fault (core dumped)

The file can be opened and a message read in, but any attempt to read the message data array using .values or .data results in a segmentation fault.

Further investigation suggests that the problem may be with the packing scheme used in these files. They use complex packing and spatial differencing (scheme 3 in grib_repack). If I use the wgrib2 utility to repack a file with another scheme (e.g., simple packing), pygrib can read the file with no errors. Additionally, if I attempt to repackage a message using code similar to what is in grib_repack, I get the following errors:

>>> rr['editionNumber']
2
>>> rr['dataRepresentationTemplateNumber']
3
>>> rr['dataRepresentationTemplateNumber'] = 0
GRIB_API ERROR   :  unable to get groupSplittingMethodUsed as long (Key/value not found)
GRIB_API ERROR   :  unable to get codedValues as double array (Key/value not found)
GRIB_API ERROR   :  unable to get values as double array (Key/value not found) Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "pygrib.pyx", line 1141, in pygrib.gribmessage.__setitem__ (pygrib.c:12685)
RuntimeError: b'Key/value not found'

This looks to me like pygrib is looking for some packing parameters that do not exist in the message.

This problem is occurring on a 64-bit Ubuntu linux machine. Has anyone else encountered this problem or have an explanation of what is different about these files?

Regards,
Rick

I am trying to install pygrib using the following pip command
#!/bin/bash
pip install --global-option=build_ext
--global-option="-I/usr/local/grib_api/include/"
--global-option=build_ext
--global-option="-L/usr/local/jasper-1.900.1/lib/ -ljasper"
--global-option=build_ext
--global-option="-L/usr/local/grib_api/lib/ -lgrib_api"
--global-option=build_ext
--global-option="-L/usr/local/libpng_1.6.18/lib/ -lpng"
pygrib

Yet, I got the following errors
gcc -pthread -shared -L/home/ahmed/anaconda3/lib -Wl,-rpath=/home/ahmed/anaconda3/lib,--no-as-needed build/temp.linux-x86_64-3.5/pygrib.o -L/opt/local/lib -L/opt/local/lib64 -L/opt/local/lib -L/opt/local/lib64 -L/opt/local/lib -L/opt/local/lib64 -L/usr/lib -L/usr/lib64 -L/usr/lib -L/usr/lib64 -L/home/ahmed/anaconda3/lib -Wl,--enable-new-dtags,-R/opt/local/lib -Wl,--enable-new-dtags,-R/opt/local/lib64 -Wl,--enable-new-dtags,-R/opt/local/lib -Wl,--enable-new-dtags,-R/opt/local/lib64 -Wl,--enable-new-dtags,-R/opt/local/lib -Wl,--enable-new-dtags,-R/opt/local/lib64 -Wl,--enable-new-dtags,-R/usr/lib -Wl,--enable-new-dtags,-R/usr/lib64 -Wl,--enable-new-dtags,-R/usr/lib -Wl,--enable-new-dtags,-R/usr/lib64 -lgrib_api -ljasper -lopenjpeg -lpng -lz -lpython3.5m -o build/lib.linux-x86_64-3.5/pygrib.cpython-35m-x86_64-linux-gnu.so
/bin/ld: cannot find -lgrib_api
/bin/ld: cannot find -ljasper
/bin/ld: cannot find -lopenjpeg
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
Any ideas !
Thanks in advance

Hello Jeffrey,

This is not really an issue, but mostly an improvement request.
I would like to have the 'step' attribute (timestep of grib message) accessible by grib selector function.
Let me explain. Here is a piece of code to extract tmax field from GFS file (ftp://ftp.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.2015031600/gfs.t00z.pgrb2.0p25.f006).

import pygrib
grib_file=pygrib.open('gfs.t00z.pgrb2.0p25.f006')
tmax_msg = grib_file.select(level=2,shortName='tmax', 'typeOfLevel': 'heightAboveGround')[0]

Here I didn't use the 'step' attribute to get the message and the extract works fine. However, I'm not sure that the selection will return only one message due to missing timestep condition (I know this is a one timestep file, but I need my code to be more generic, and some ECMWF files contain multiple timesteps). So I would like to be more specific in my selection map. The step attribute seems to appear in message field :

print(tmax_msg['step'])
6

But if I add this attribute to filter, I get a data not found exception:

tmax_msg = grib_file.select(level=2,shortName='tmax', 'typeOfLevel': 'heightAboveGround', step=6)[0]
Traceback (most recent call last):
File "", line 1, in
File "pygrib.pyx", line 609, in pygrib.open.select (pygrib.c:5851)
ValueError: no matches found

Apparently, only the 'stepRange' attribute can be used to filter over timesteps, but it is quite annoying because this field depends on parameter :
tmax -> stepRange = '0-6'
2t -> stepRange = '6'

Do you think it could be possible to use the 'step' attribute as a common timestep filter for all parameter ?

Best regards

Julien

From [email protected] on March 09, 2010 23:12:19

What steps will reproduce the problem? 1. Download any GRIB2 file from ftp://ftp.ssec.wisc.edu/pub/eosdb/dbcras/ ,
e.g. ftp://ftp.ssec.wisc.edu/pub/eosdb/dbcras/2010_03_08_067/08Mar10.12z.dbCRAS.SouthAfrica.grib2 2. Run the attached test.py script (using 08Mar10.12z.dbCRAS.SouthAfrica.grib2)

The script aborted with the following output:
1:Surface pressure:Pa (instant):polar_stereographic:surface:level 0:fcst
time 0:from 201003081200
Traceback (most recent call last):
File "test.py", line 4, in
for grb in grbs:
File "pygrib.pyx", line 289, in pygrib.open.next (pygrib.c:1606)
RuntimeError: Unknow error -145657936

The same data file can be decoded without any error using 'wgrib2'.

I am using pygrib-1.1 compiled from source, with grib_api-1.8
running Ubuntu 9.10

Attachment: test.py

Original issue: http://code.google.com/p/pygrib/issues/detail?id=2

I have a grib that only contains the value 100.0 which appears not to unpack correctly, the values unpack to 1000.0 not 100.

Sample file containing only the value 100.0 for each cell in grid:
http://yacht.net.nz/~ckd/cldd23h15.grb

Same grid with multiple values that is handled correctly:
http://yacht.net.nz/~ckd/cldd23h17.grb

Other tools (eg. CDO) handle the file with homogeneous values correctly.

>>> gs = pygrib.open('cldd23h15.grb')
>>> g = gs[1]
>>> g
1:Total Cloud Cover:% (instant):regular_ll:entireAtmosphere:level 0:fcst time 3 :from 201603231200
>>> g.values
array([[ 1000.,  1000.,  1000., ...,  1000.,  1000.,  1000.],
       [ 1000.,  1000.,  1000., ...,  1000.,  1000.,  1000.],
       [ 1000.,  1000.,  1000., ...,  1000.,  1000.,  1000.],
       ...,
       [ 1000.,  1000.,  1000., ...,  1000.,  1000.,  1000.],
       [ 1000.,  1000.,  1000., ...,  1000.,  1000.,  1000.],
       [ 1000.,  1000.,  1000., ...,  1000.,  1000.,  1000.]])
>>> gs = pygrib.open('cldd23h17.grb')
>>> g = gs[1]
>>> g
1:Total Cloud Cover:% (instant):regular_ll:entireAtmosphere:level 0:fcst time 5 :from 201603231200
>>> g.values
array([[ 100. ,  100. ,  100. , ...,  100. ,  100. ,  100. ],
       [ 100. ,  100. ,  100. , ...,  100. ,  100. ,  100. ],
       [ 100. ,  100. ,  100. , ...,  100. ,  100. ,  100. ],
       ..., 
       [ 100. ,   99.9,   99.9, ...,  100. ,  100. ,  100. ],
       [  99.9,   99.9,   99.9, ...,  100. ,  100. ,  100. ],
       [  99.9,   99.9,   99.8, ...,  100. ,  100. ,  100. ]])

Using pygrib 2.0.0 and python 2.7 on ubuntu 14.04.

Any Ideas?