Avoid using h5netcdf? about metoffice_ec2 HOT 9 CLOSED

Eak, sorry about this, I really need to write some more tests so we can catch simple failures like this before pushing to master. Sorry, my bad! I'll look into this next week

from metoffice_ec2.

@JackKelly Will do! Thanks.

from metoffice_ec2.

It's possible that we can just remove engine='h5netcdf'. Or see here for an alternative approach: pydata/xarray#1075 (comment)

from metoffice_ec2.

Getting a lot of these errors atm:

2020-05-29 17:49:47,688 - metoffice_ec2 - ERROR - No module named 'h5netcdf'
Traceback (most recent call last):
  File "scripts/ec2.py", line 114, in loop
    load_subset_and_save_data(mo_message, s3)
  File "scripts/ec2.py", line 64, in load_subset_and_save_data
    dataset = mo_message.load_netcdf()
  File "/usr/src/app/metoffice_ec2/message.py", line 73, in load_netcdf
    return xr.open_dataset(netcdf_bytes_io)
  File "/opt/conda/envs/metoffice_ec2/lib/python3.8/site-packages/xarray/backends/api.py", line 533, in open_dataset
    store = backends.H5NetCDFStore.open(
  File "/opt/conda/envs/metoffice_ec2/lib/python3.8/site-packages/xarray/backends/h5netcdf_.py", line 123, in open
    import h5netcdf
ModuleNotFoundError: No module named 'h5netcdf'

Reopening this issue, didn't seem to resolve it.

from metoffice_ec2.

Reverted this via #30 to fix deployment.

from metoffice_ec2.

No worries! Yes, we should probably add some tests haha

from metoffice_ec2.

I've recreated this issue locally.

I've tried changing the engine to all the options (other than h5netcdf) and it always responds: ValueError: can only read bytes or file-like objects with engine='scipy' or 'h5netcdf'

from metoffice_ec2.

I've recreated this issue locally.

I've tried changing the engine to all the options (other than h5netcdf) and it always responds: ValueError: can only read bytes or file-like objects with engine='scipy' or 'h5netcdf'

But I think I've got it working without h5netcdf using the netCDF4 trick described here: pydata/xarray#1075 (comment)

@FWirtz no rush, but please could you try this new code on AWS?

from metoffice_ec2.

That worked!

from metoffice_ec2.