changliao1025 / pyhexwatershed Goto Github PK

The Python interface to HexWatershed a mesh independent flow direction model for hydrologic models

License: Other

Python 97.13% Makefile 2.36% Dockerfile 0.32% Batchfile 0.10% Shell 0.09%

visualisation hydrology watershed gis flow-direction flow-routing graph unstructured-meshes river-networks terrain-analysis

pyhexwatershed's Introduction

`HexWatershed`

HexWatershed: a mesh independent flow direction model for hydrological models.

This Python package provides a Python interface to the underlying HexWatershed model.

HexWatershed has been compiled and tested on various 64-bit Linux and Mac based platforms.

`Installation`

Ensure you have a c++ compiler and the cmake utility installed.

Build the HexWatershed C++ backend and place the binary under the bin path.

Install the python package using conda: conda install -c conda-forge hexwatershed

For detailed installation instruction, please refer to the documentation.

`Usage`

Please use this tutorial to run the examples.

`Acknowledgement`

This work was supported by the Earth System Model Development program areas of the U.S. Department of Energy, Office of Science, Office of Biological and Environmental Research as part of the multi-program, collaborative Integrated Coastal Modeling (ICoM) project.

`License`

Battelle Memorial Institute (hereinafter Battelle) hereby grants permission to any person or entity lawfully obtaining a copy of this software and associated documentation files (hereinafter “the Software”) to redistribute and use the Software in source and binary forms, with or without modification. Such person or entity may use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and may permit others to do so, subject to the following conditions:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimers.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Other than as used herein, neither the name Battelle Memorial Institute or Battelle may be used in any form whatsoever without the express written consent of Battelle.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL BATTELLE OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

`References`

There are a number of publications that describe the algorithms used in HexWatershed in detail. If you make use of HexWatershed in your work, please consider including a reference to the following:

`Scientific Publications`

Liao, Chang, Zhou, T., Xu, D., Tan, Z., Bisht, G., Cooper, M. G., et al. (2023). Topological relationship-based flow direction modeling: Stream burning and depression filling. Journal of Advances in Modeling Earth Systems, 15, e2022MS003487. https://doi.org/10.1029/2022MS003487
Liao, Chang, Zhou, T., Xu, D., Cooper, M. G., Engwirda, D., Li, H.-Y., Leung, L. R. (2023). Topological relationship-based flow direction modeling: Mesh-independent river networks representation. Journal of Advances in Modeling Earth Systems, 15, e2022MS003089. https://doi.org/10.1029/2022MS003089
Liao, Chang, Tian Zhou, Donghui Xu, Richard Barnes, Gautam Bisht, Hong-Yi Li, Zeli Tan, et al. (02/2022AD) 2022. “Advances In Hexagon Mesh-Based Flow Direction Modeling”. Advances In Water Resources 160. Elsevier BV: 104099. https://doi.org/10.1016/j.advwatres.2021.104099.
Liao, C., Tesfa, T., Duan, Z., & Leung, L. R. (2020). Watershed delineation on a hexagonal mesh grid. Environmental Modelling & Software, 128, 104702. https://doi.org/10.1016/j.envsoft.2020.104702

`Software`

Liao, Chang, Cooper, M. G. PyFlowline a mesh independent river network generator for hydrologic models. Journal of Open Source Software. https://doi.org/10.21105/joss.05446
Liao. C. (2022) Pyflowline: a mesh independent river network generator for hydrologic models. Zenodo. https://doi.org/10.5281/zenodo.6407299
Liao. C. (2022). HexWatershed: a mesh independent flow direction model for hydrologic models (0.1.1). Zenodo. https://doi.org/10.5281/zenodo.6425881

For a full list of references including applications of HexWatershed in other projects, please refer to the documentation.

pyhexwatershed's People

Contributors

Stargazers

Watchers

Forkers

ecustwy waveletswave mgcooper

pyhexwatershed's Issues

HPC job function

Create a slurm job file through the Python API.
Allow the bash script to run the Python script and run the C++ component within Python.

Setup a slurm job file for the workflow

Option 1: connect all steps into one single python script and let the python script run the hexwatershed.
Option 2: separate different steps and run them in a sequence.

Spatial reference

The spatial reference is used for mesh generation. Currently, the DEM has the spatial reference already, so is it necessary to use an additional vector file?

Configuration file update

The configuration file will be updated after the pyflowline simulation.
changliao1025/pyflowline#31
However, should this be done by the main program or the sub-program is not decided.
The main configuration is passed to pyflowline, so there are essentially the same.
It is more straightforward to let pyflowline hand this because the basin config is within pyflowline.

Variable name across different modules

Currently, the existing modules within the HexWatershed can have different names in different settings.
I will update each component so they will be more consistent.

Add the advanced plot function using new json

Add major arguments into the read function

I will add additional optionally arguments in the read function to make it easier to set up a case without modifying the configuration JSON file.

Vertex location information

While it is desirable to have vertex location, it will incur additional storage and file I/O burden.
The vtk function can be supported in both C++ and Python.
The other visualization can be designed within Python.

Update how hexwatershed setup pyflowline

There is a dependency relationship, but we need a better way to setup both models.

Use cell center distance for slope calculation

Boundary processing

Image processing was used in hexwatershed global scale simulation:
globe_hexagon_erossion ();
globe_hexagon_dilation ();

The reason for these two operations is to ensure each boundary cell ONLY has two neighboring boundary cells.

In the latest development, they will be removed because the requirement will be dropped. We will a different approach to build the queue.

DEM source

Since MPAS mesh provides elevation, it will be used in global scale.
Elevation profile requires fine scale DEM, which is not available yet in MPAS netcdf file.

A hybrid topology based method for different steps

The model will support the PyFlowline to be topology-based while the stream burning is non-topology-based.

Boundary or flowline extent

To create mesh from scratch, the program needs to know the extent (up, top, left, right).
This information can be retrieved from spatial extend of both DEM and flowline.

To have consistent design, we will use the DEM as primary information source, and also use flowline as secondary information.

Map projection is another issue.

A list of attribute output

Drainage area
TWI
Travel length
River slope profile

DEM for other meshes

Add DEM resample back to structured meshes.

GDAL create mesh from raster

GDAL intercept two shapefiles

After the recent updates, the GeoJSON is used for all data transfer.
The intersection is done using the GDAL python API.

Use elevation profile for slope model

This requires that the depression filling to be updated.

Auto update boundary using the outlet location

Currently, if the mesh was generated "well", then the outlet location may not be on the edge, which causes the program to crash in other meshes.
In the next development, we will add a function to auto-generate mesh that will match with outlet location and DEM.

Stream segment output

The backend will now output a json file which contains all the stream segments for each watershed.

Example folder re-organization

Following the other repos, the main repo now uses a new folder structure for different domain under the example folder.

Checkpoint feature

Running a simulation with all the steps or features may be computationally expensive, adding a checkpoint feature may speed up some process.

Negative slope in the outlet cell

This bug is due to a reversed slope assignment.

Add a threshold for river bed slope

This threshold should consider ocean's impacts on delta.

Stream flowline preparation

#step 1: convert it to json format
#step 2: merge all as one single feature   
#step 3: split into segment   
#step 4: remove loops
#step 5: remove small headwater segment
#step 6: intersect with mesh and simplify
#step 7: rebuild index and order
#step 8: calculate properties

Vertex comparison

There is a bug in the existing vertex comparison algorithm.
Although the algorithm avoids using a threshold to compare two objects, the precision of float can cause this bug in rare cases.

Flowline simplification process

The simplification is required, however this can be done in either Python or in C++.
There are both pros and cons.
Reason to pick Python:
Easy to test and implement
Should be fast enough because information is limited
There are some existing C++ to Python conversion tools.
Reason to not pick Python:
Not sure about the STL alternative.
Global scale high resolution computational demand.

Reason to pick C++:
There are existing algorithm built;
performance;
natural C++ STL library
Reason not to pick C++:
Not ideal in long run.

changliao1025 / pyhexwatershed Goto Github PK

pyhexwatershed's Introduction

HexWatershed

Installation

Usage

Acknowledgement

License

References

Scientific Publications

Software