powellb / seapy Goto Github PK

View Code? Open in Web Editor NEW

28.0 9.0 20.0 2 MB

State Estimation and Analysis in Python

License: MIT License

Python 71.18% Fortran 28.82%

oceanography roms model analysis grid numpy

seapy's Introduction

State Estimation and Analysis in PYthon (SEAPY)

Tools for working with ocean models and data.

SEAPY requires: basemap, h5py, joblib, netcdf4, numpy, numpy_groupies, rich and scipy.

Installation

Install from Conda-Forge

Install from conda-forge with the Conda package manager:

$ conda install -c conda-forge seapy

You should also consider making conda-forge your default channel. See the conda-forge tips and tricks page.

The Conda-Forge SEAPY feedstock is maintained by Filipe Fernandes, ocefpaf. As of February 2021 there are binary packages on all the platforms that Conda-Forge supports: Python 3.6 through 3.9 on Linux, Windows and Mac OSX (all 64-bit).

To remove seapy:

$ conda remove seapy

Install from PyPI

Install from PyPI with PIP:

$ pip install seapy-ocean

Note that on PyPI (but nowhere else) the package name is seapy-ocean to avoid a name clash with another package. The module name is still seapy.

SEAPY packages on PyPI have been built and uploaded by Mark Hadfield, hadfieldnz. There is a source distribution that should build with no problems on Linux (and Mac OSX, but we haven't tested it). In the pst there have been binary distributions for Windows (64-bit), but these have now been deleted as binary builds with PIP are no longer supported.

In a Conda environment, it is quite possible to install with PIP, but dependency handling and updating will be cleaner if you use the Conda package.

To remove seapy-ocean

$ pip uninstall seapy-ocean

Install from source code on GitHub.com

The SEAPY source code is maintained by Brian Powell, (powellb)[https://github.com/powellb]. Releases are made on the master branch

Install from GitHub.com with PIP:

$ pip install git+git://github.com/powellb/seapy@master

OR clone a copy of the source and install in editable mode, eg:

$ git clone https://github.com/powellb/seapy.git
$ pip install -e seapy

With an editable-mode installation, changes you make to your copy of the source code will take effect when you import the module.

In principle it is possible to build from source on Windows--and success with this has been achieved in the past--but the process tends to break with changes in the environment or Python version, so we don't recommend it or support it.

Contributing

If you've installed from source in editable mode, then you should definitely consider forking your own copy of the repository. This allows you to keep your changes under revision control on GitHub.com and potentially contribute them to the main project. You should follow the procedures described in this Git Workflow document.

Forking on GitHub.com is a lightweight process that won't complicate your workflow and keeps the relationship between your work and the original project clear, so it is strongly advised to do it early. However the immutable and unique nature of Git commits means that you can create and populate a fork later if you want to, as long as you have saved your work somewhere in Git format. To create a fork you will need a GitHub.com user account.

All your changes should be committed to a branch other than "master", which is reserved for the master branch in Brian Powell's repository (or copies thereof). A common practice in the existing SEAPY forks is to use a branch name matching your user name for your own work. However if you are developing a specific feature or bug fix to be pulled into master, it may be sensible to name the branch after that feature or bug fix.

Examples

Many of the time-saving features are in generating fields for running the ROMS model.

To load the meta information about a model (ROMS, HYCOM, MITgcm, POM, SODA), load an output file (history, average, climatology, grid, etc.) via:

 >> mygrid = seapy.model.asgrid(filename)

 >> mygrid
 C-Grid: 32x194x294

 >> print(mygrid)
 filename
 32x194x294: C-Grid with S-level
 Available: I,J,_isroms,_nc,angle,cgrid,cs_r,depth_rho,depth_u,depth_v,dm,dn,eta_rho,eta_u,eta_v,f,filename,h,hc,lat_rho,lat_u,lat_v,lm,ln,lon_rho,lon_u,lon_v,mask_rho,mask_u,mask_v,n,name,pm,pn,s_rho,shape,spatial_dims,tcline,theta_b,theta_s,thick_rho,thick_u,thick_v,vstretching,vtransform,xi_rho,xi_u,xi_v

Most methods available in SEAPY require a grid, which can be specified as a "filename" or as a grid object.

Find out how to download global HYCOM data that will span my grid from 1/1/2015 through 5/1/2015:

 >> seapy.model.hycom.load_history("hycom_file.nc", start_time=datetime(2015,1,1),
                                  end_time=datetime(2015,5,1),
                                  grid=mygrid, load_data=False)
 ncks -v water_temp,salinity,surf_el,water_v,water_u -d time,352,352 -d lat,1204,1309 -d lon,2438,2603 http://tds.hycom.org/thredds/dodsC/GLBu0.08/expt_91.1 hycom_file.nc

This will display the 'ncks' command necessary to download the data. If you want to have SEAPY download it (not recommended due to server-speed), use 'load_data=True'.

Once you have HYCOM data, interpolate it to your grid

 >> seapy.roms.interp.to_clim("hycom_file.nc", "my_clim.nc",
                   dest_grid=mygrid, nx=1/6, ny=1/6,
                   vmap={"surf_el":"zeta", "water_temp":"temp",
                   "water_u":"u", "water_v":"v", "salinity":"salt"})

Generate boundary conditions from the climatology

 >> seapy.roms.boundary.from_roms("my_clim.nc", "my_bry.nc")

Generate initial conditions from the climatology

 >> seapy.roms.initial.from_roms("my_clim.nc", "my_ini.nc")

You now have what you need to run your model

To set up data assimilation, download the raw observations (e.g., aviso_map_day1.nc, aviso_map_day2.nc, argo_day1.nc ). You can then process the data:

 >> dt = 400/86400       # time-step of the model in days
 >> aviso_gen = seapy.roms.obsgen.aviso_sla_map(mygrid, dt)
 >> aviso_gen.batch_files(seapy.list_files('./aviso.*nc'), 'aviso_roms_#.nc')
 >> argo_gen = seapy.roms.obsgen.argo_ctd(mygrid, dt)
 >> obs = argo_gen.convert_file("argo_day1.nc")
 >> obs.to_netcdf("argo_roms_1.nc")

Put all of the processed observations files together into a file for a given assimilation window

 >> seapy.roms.obs.merge_files(seapy.list_files('.*roms_[0-9]+.nc'), 'roms_obs_#.nc', np.arange([0, 10.1, 5]))

There are many more things that can be done, but these show some of the power available via simple commands.

seapy's People

Contributors

Stargazers

Watchers

Forkers

dalepartridge ivicajan rsignell-usgs ocefpaf mnkmishra02 hadfieldnz sciencewiki joaometocean ivand890 nilodna lijunde metocean samanthastevenson laspg huyquangtranaus kevinrosa andressepulveda

seapy's Issues

The exit codes of the workers are {SIGSEGV (-11)}

Hi, I'm currently using "seapy" to create boundaries, meteorology, and initial data for the ROMS model.
So I was working with reference to README.md,
At the stage of "4. Once you have HYCOM data, interpolate it to your grid", the following error occurs and I am in trouble.

seapy.roms.interp.to_clim("hycom_file.nc", "my_clim.nc", dest_grid=mygrid, nx=1/6, ny=1/6, vmap={"surf_el":"zeta", "water_temp":"temp", "water_u":"u", "water_v":"v", "salinity":"salt"})

Output

"joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The exit codes of the workers are {SIGSEGV (-11)} "

I think "seapy /seapy/roms/interp.py" is involved.

Can you tell me what to do?
thank you.

obs.x (and all others) wtr to python start 0 and fortan (ROMS) start 1?

Brian & Dale,
I am puzzled as when you call seapy gridder and use grid.ij (or grid.ijk) you have all indexes starting from 0 (as python counts)/ You need that when searching in the matrices etc. However, when putting all into netcdf we have to follow fortran which use start from 1. Is it possible that seapy created obs.x, .y, .z which are -1 from the one created before with matlab's obs_gridder? Should we in to_netcdf add obs.x += 1 (the same for y)?
Those are the only indexes, others are real locations, values...

Cheers,
Ivica

Extreme momentum values created in boundary file

I've created a climatology file using seapy as:

seapy.roms.interp.to_clim(base_dir + "hycom_file.nc",
                          dest_file=base_dir + "my_clim.nc",
                          dest_grid=mygrid, nx=1/25., ny=1/25.,
                          vmap={"zeta":"zeta", "temp":"temp",
                          "u":"u", "v":"v", "salt":"salt"})

Strangely, an ncdump of the climatology file reveals the first couple of lines of data (near the masked data) for ubar and v_bar have some extreme values (e.g., -54,000). Does anyone have any idea of what I might be doing wrong?

Question about observation netcdf file

Greetings

Recently our group switched from matlab to python using seapy package
for the pre-process of the observations that are need for ROMS 4DVAR
scheme (formerly PSAS now RBL4DVAR) . I am experiencing a weird
issue with the python processed observation file when i use it: During
the initialization of the first inner loop (outer = 001, inner =001 not 000)
although obs are not rejected ROMS doesn't read their values for the
first 12 hours and this - let's say - gap ends (obviously) to a blowup. Using
matlab processed file with the same datasets for observations model runs smoothly.
obs_error and obs_variance differ between the two files - I am setting
fixed error values for the state variables in the matlab scripts - but also
the number of observations is different, with the seapy produced file having
more compared to matlab's.

My experience with python is limited (now I am learning) and I am trying to understand
why this is happening but I had no luck. Any help or insight will be highly appreciated.

I also am using the matlab scripts from https://www.myroms.org/svn/src/matlab to create the
observation file.

Kind Regards

Giannis Mamoutos

Please create a release

@powellb , to facilitate @ocefpaf building a conda-forge package, could you guys tag a release?

Could even be 0.1.0 or something...

tide.py and new numpy

in tide.py you should add option for pickle to be compatable with numpy 1.17+
line 25 I think:
with np.load(os.path.dirname(file) + "/constituents.npz", allow_pickle=True) as data:

otherwise install via conda is not working (if having updated numpy)

Cheers
Ivica

Load HYCOM data issues

Hello, everyone
I am currently using SEAPy to make ROMS initial field and boundary field files. But when I run seapy.model.hycom.load_history("hycom_file.nc", start_time=datetime.datetime(2019,8,1), end_time=datetime.datetime(2019,8,31), grid=mygrid, url='http://tds.hycom.org/thredds/dodsC/GLBu0.08/expt_91.1', load_data=True).
Exception: Cannot find valid times.How da I solve this problem?
In addition I also have other problem is that run mygrid = seapy.model.asgrid('roms_sea.nc') warn (" could not compute grid depths. "). Will this error not affect my subsequent interpolation?

seapy/lib.py

should have in line 25
_default_timeref = "days since " + default_epoch.strftime("%Y-%m-%d %H:%M:%S")
instead
_default_timeref = "days since " + default_epoch.strftime("%Y-%m-%m %H:%M:%S")

You have %m and should be %d for day ...
Ivica

P.S. Not sure if you have it on other places as well

Seapy superobs not considering ROMS cell-edges!?

Hi,

It appears that Seapy is not considering the C-staggering locations:

A bigger view:

Clearly, the code is superobing where it shouldn't. Looks like the cell edges are not taking into account. An offset is missing by the above behaviour.

raw data source: JPLMUR
pseudo code:

 file=mur_file
 otype = seapy.roms.obsgen.ostia_sst_map(grid_file, grid_dt)
 obs = otype.convert_file(file)
 obs.to_netcdf(outfile)

Build issues

Tried building as instructed with make all, but build failed -- couldn't find oalib.so.

Tried doing cd src; make oalib, which gave warnings:

oalib.F:832.28:

      call inverse(Gp,c,c,A,issing)
                            1
Warning: Type mismatch in argument 'issing' at (1); passed INTEGER(4) to REAL(8)
oalib.F:1114.28:

and failed to copy the existing oalib.so because it didn't get picked up by the regex:

cp oalib.*.so ../oalib.so
cp: cannot stat `oalib.*.so': No such file or directory

Same issue for hindices

Import of setup duplicated in setup.py

In setup.py on the master branch at lines 6 & 7, the setup function is imported twice.

from setuptools import find_packages, setup
from numpy.distutils.core import setup

The effect of the second import is to replace the first. This gives the correct behaviour (as numpy.distutils.core is the one required) but is potentially confusing. I recommend

from setuptools import find_packages
from numpy.distutils.core import setup

Priority is low as there is no change in function. I have not prepared a PR.

No module named 'seapy.external'

Hi,

Is there any suggestion to fix the following error message:

"/home/hugo/Pyroms_roms/seapy/seapy/oa.py", line 13, in
from seapy.external import oalib
ModuleNotFoundError: No module named 'seapy.external'

Thanks
Huy

Segmentation fault in seapy.roms.interp.__interp_grids lines 261-314

I am trying to create a climatology file in seapy using a grid created using pyroms. I am getting a segmentation fault using Python 3.5.1 on OS X El Cap. I have traced the error as far as I can, and it appears to be coming from lines 261-314 in interp.py. It may be occurring in __interp2_thread or interp3_thread, but I am unfamiliar with joblib to troubleshoot any further. Can anybody else reproduce this? It might be caused by an error in my grid, but I thought I'd share the segmentation fault anyway. On a side note... Are there any plans for a Python 2.x version? I would love to be able to use this side-by-side with my pyroms installation. The output I am getting is below is repeated several times before Python quits. Thanks.

/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/numpy/lib/shape_base.py:873: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future return c.reshape(shape_out) /Users/redacted/src/seapy/roms/interp.py:90: UserWarning: nx or ny values are too large for stable OA, 71.000000 warn("nx or ny values are too large for stable OA, {:f}".format(ksize)) /Users/redacted/src/seapy/lib.py:195: MaskedArrayFutureWarning: setting an item on a masked array which has a shared mask will not copy the mask and also change the original mask array in the future. Check the NumPy 1.11 release notes for more information. fld[lst] = nfld[lst] / count[lst]