ch-earth / datatool Goto Github PK

datatool: Dynamical Data Processing Tool for HPCs

License: GNU General Public License v3.0

Shell 96.78% NCL 3.22%

datatool's Introduction

Description

This repository contains scripts to process meteorological datasets in NetCDF file format. The general usage of the script (i.e., ./extract-dataset.sh) is as follows:

Usage:
  extract-dataset [options...]

Script options:
  -d, --dataset                     Meteorological forcing dataset of interest
  -i, --dataset-dir=DIR             The source path of the dataset file(s)
  -v, --variable=var1[,var2[...]]   Variables to process
  -o, --output-dir=DIR              Writes processed files to DIR
  -s, --start-date=DATE             The start date of the data
  -e, --end-date=DATE               The end date of the data
  -l, --lat-lims=REAL,REAL          Latitude's upper and lower bounds;
                                    optional; within the [-90, +90] limits
  -n, --lon-lims=REAL,REAL          Longitude's upper and lower bounds;
                                    optional; within the [-180, +180] limits
  -a, --shape-file=PATH             Path to the ESRI shapefile; optional
  -m, --ensemble=ens1,[ens2,[...]]  Ensemble members to process; optional
                                    Leave empty to extract all ensemble members
  -M, --model=model1,[model2,[...]] Models that are part of a dataset,
                                    only applicable to climate datasets, optional
  -S, --scenario=scn1,[scn2,[...]]  Climate scenarios to process, only applicable
                                    to climate datasets, optional
  -j, --submit-job                  Submit the data extraction process as a job
                                    on the SLURM system; optional
  -k, --no-chunk                    No parallelization, recommended for small domains
  -p, --prefix=STR                  Prefix  prepended to the output files
  -b, --parsable                    Parsable SLURM message mainly used
                                    for chained job submissions
  -c, --cache=DIR                   Path of the cache directory; optional
  -E, [email protected]      E-mail user when job starts, ends, or               
                                    fails; optional
  -u, --account                     Digital Research Alliance of Canada's sponsor's
                                    account name; optional, defaults to 'rpp-kshook' 
  -L, --list-datasets               List all the available datasets and the
                                    corresponding keywords for '--dataset' option
  -V, --version                     Show version 
  -h, --help                        Show this screen and exit

Available Datasets

#	Dataset	Time Period	DOI	Description
1	GWF-NCAR WRF-CONUS I	Hourly (Oct 2000 - Dec 2013)	10.1007/s00382-016-3327-9	link
2	GWF-NCAR WRF-CONUS II¹	Hourly (Jan 1995 - Dec 2015)	10.5065/49SN-8E08	link
3	ECMWF ERA5²	Hourly (Jan 1950 - Dec 2020)	10.24381/cds.adbb2d47 and link	link
4	ECCC RDRSv2.1	Hourly (Jan 1980 - Dec 2018)	10.5194/hess-25-4917-2021	link
5	CCRN CanRCM4-WFDEI-GEM-CaPA	3-Hourly (Jan 1951 - Dec 2100)	10.5194/essd-12-629-2020	link
6	CCRN WFDEI-GEM-CaPA	3-Hourly (Jan 1979 - Dec 2016)	10.20383/101.0111	link
7	ORNL Daymet	Daily (Jan 1980 - Dec 2022)³	10.3334/ORNLDAAC/2129	link
8	Alberta Gov Climate Dataset	Daily (Jan 1950 - Dec 2100)	10.5194/hess-23-5151-201	link
9	Ouranos ESPO-G6-R2	Daily (Jan 1950 - Dec 2100)	10.1038/s41597-023-02855-z	link
10	Ouranos MRCC5-CMIP6	hourly (Jan 1950 - Dec 2100)	TBD	link
11	NASA NEX-GDDP-CMIP6	Daily (Jan 1950 - Dec 2100)	10.1038/s41597-022-01393-4	link

General Example

As an example, follow the code block below. Please remember that you MUST have access to Digital Research Alliance of Canada (DRA) clusters (specifically Graham) and have access to RDRSv2.1 model outputs. Also, remember to generate a Personal Access Token with GitHub in advance. Enter the following codes in your Graham shell as a test case:

foo@bar:~$ git clone https://github.com/kasra-keshavarz/datatool # clone the repository
foo@bar:~$ cd ./datatool/ # move to the repository's directory
foo@bar:~$ ./extract-dataset.sh -h # view the usage message
foo@bar:~$ ./extract-dataset.sh  \
  --dataset="rdrs" \
  --dataset-dir="/project/rpp-kshook/Climate_Forcing_Data/meteorological-data/rdrsv2.1" \
  --output-dir="$HOME/scratch/rdrs_outputs/" \
  --start-date="2001-01-01 00:00:00" \
  --end-date="2001-12-31 23:00:00" \
  --lat-lims=49,51  \
  --lon-lims=-117,-115 \
  --variable="RDRS_v2.1_A_PR0_SFC,RDRS_v2.1_P_HU_09944" \
  --prefix="testing_";

See the examples directory for real-world scripts for each meteorological dataset included in this repository.

Logs

The datasets logs are generated under the $HOME/.datatool directory, only in cases where jobs are submitted to clusters' schedulers. If processing is not submitted as a job, then the logs are printed on screen.

New Datasets

If you are considering any new dataset to be added to the data repository, and subsequently the associated scripts added here, you can open a new ticket on the Issues tab of the current repository. Or, you can make a Pull Request on this repository with your own script.

Support

Please open a new ticket on the Issues tab of this repository for support.

License

Meteorological Data Processing Workflow - datatool
Copyright (C) 2022-2023, University of Saskatchewan
Copyright (C) 2023-2024, University of Calgary

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.

For access to the files on Graham cluster, please contact Stephen O'Hearn. ↩
ERA5 data from 1950-1979 are based on ERA5 preliminary extenion and 1979 onwards are based on ERA5 1979-present. ↩
For the Peurto Rico domain of the dataset, data are available from January 1950 until December 2022. ↩

datatool's People

Contributors

Stargazers

Watchers

Forkers

wijayard shervangharari adrien63ro h294liu

datatool's Issues

Issues with the bottom pixels of `RDRSv2.1`

The 40 bottom pixels in RDRSv2.1 are not valid and should be removed from further evaluations - the following is the text received from ECCC:

While doing some verifications of the RDRS_v2.1 product, ECCC has found out that there is a discontinuity at the very south of the domain (see figure attached). This map represents the 24-h precipitation accumulated from April to May 2014 (from the CaPA offline reanalysis v2.1). This is due to differences in spatial resolution of products used to produce RDRS (GDPS/RDPS). Please consider not using the most southern 40 pixels of the current RDRS_v2.1 product. These grid cells will be removed from any further RDRS product, but current ones will not be post-processed.

`ANUSPLIN` to be added to the list of dataset

Problem

The ANUSPLIN dataset could be added to the list of dataset.

Extra information

The dataset information is available here: https://cfs.nrcan.gc.ca/projects/3/. However, the dataset file is missing and should be found elsewhere.

Consistent data repositories across HPCs

The data repository need to become consistent in order to be sharable across various HPCs. Data repository is not directly included in here, however, path names need to be updated soon.

Some recommendations:

lower case directory names,
download workflows,
TBD

`rdrsv2.1` sometimes does not crop to the full extent of the input Shapefile

Sometimes the cropped RDRSv2.1 grid cells to not entirely cover the requested domain. An example is below:

As a quick solution, a value of 0.1 degree could be added/subtracted given the maximum/minimum latitude and longitude values. This is a quick solution and a better one needs to be implemented to crop the extents seamlessly.

list of available variables for each dataset

Issue description

It would be nice if the list of variables are accessible from the scripts, so the users don't have to look them up online, are access a single file manually.

out-of-range dates are causing issues with `rdrsv2.1`

The tool cannot take care of out-of-range dates in processing rdrsv2.1 files. However, it seems that anything later than the --end-date is taken care, while those before in the --start-date throw an issue like the following:

ncks: ERROR file /gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/temp/rdrs/1950/1950010112.nc not found. It does not exist on the local filesystem, nor does it match remote filename patterns (e.g., http://foo/ or foo.bar.edu:file).

Since the subsetting is done in a until ... do loop, it may continue indefinitely, without any solution.

Reported-by: Wouter Knoben @wknoben

Near surface meteorological variables from 1979 to 2019 derived from bias-corrected reanalysis

Dataset

Name: Near surface meteorological variables from 1979 to 2019 derived from bias-corrected reanalysis
Description: The following dataset could be added to datatool: https://cds.climate.copernicus.eu/cdsapp#!/dataset/10.24381/cds.20d54e34?tab=form

@fuadyassin

`RDRSv2.1` precipitation unit is wrong

It is in m/hour not mm/hour

Adding Wouter Knoben's contribution to the ERA5 downloads

Issue description

The download scripts available at here need to acknowledge Wouter Knoben's contributions including parts of the CWARHM workflow.

`--account` option explanation needs `=STR`

Basically the title, this has been missing and was reported by Francesca Bearzot (not on GitHub yet).

Missing land cover types information for the Landsat NALCMS dataset

problem statement

The landcover type information for the Landsat NALCMS dataset is missing, although there is a link that directs users to the main website of the dataset. This issue has been previously brought up by @wijayard for the MODIS dataset of this repository.

`RDRSv2.1` processing script could be more efficient

Reported by: Sujata Budhathoki @sujata91

The process time for RDRSv2.1 is inefficient as it takes nearly 3 hours for any domain. This could be much more efficient and be done in a few minutes for any spatial/temporal domain of interest.

Solutions that come to mind:

first subsetting needs to be done and then the modifications on the lon value take place,
spatial subsetting to be done using ncks index values, rather than cdo's sellonlatbox method.

List of ensemble member from `ESPO-G6-R2` and `NEX-GDDP-CMIP6`

The list of ensemble members from the aforementioned datasets are missing in the documentations. This needs to be addressed for the sake of end-users.

The `--end-date` for `daymet` dataset is not functioning properly

The --end-date does not subset up to the day, but rather, looks at the yearly file. This needs to be addressed.

Parallelizing based on `variables`

Problem statement

Parallelizing based on variables could be another feature that could speed up the process. This could be added to future version.

Common functions need to be exposed globally

The general common functions need to be exposed, so other scripts throughout the package can benefit from this.

`canrcm4_wfdei_gem_capa` example not working

The example provided for canrcm4_wfdei_gem_capa dataset hangs on log in node, while others, such as wrf-i, works fine.
It would be great if you can take a look at this @kasra-keshavarz.
Thank you,

`RDRSv2.1` needs a few warnings as the file/directory structure and containing data are counterintuitive

Reporte by: @wknoben

Since the time steps of each dataset file is a bit weird (starting from 12PM of the file date and ending in date+1 1PM), it makes sense to include these on its documentation under its repository.

Also, one more note is that December 31st, 2018 file (2018123112.nc) is non-existent, so this could be added to the docs as well.

Reflecting `Available Datasets` info under ./scripts directory

The contents of the main README.md file, Available Datasets header, should be reflected under the ./scripts directory of the repository.

Adding a buffer for `datatool` spatial extents when a `--shape-file` argument is provided

It would be helpful if a buffer zone (in degrees) could be provided to the tool (with a default value of dataset resolutions) so the outputs can have slightly bigger (or smaller) extent of the boundaries of the --shape-file=PATH object.

Reported by: @wknoben and @ShervanGharari

Script and documentation of `AMES NEX-GDDP-CMIP6` dataset

Problem

The AMES NEX-GDDP-CMIP6 dataset needs to be added to the list of scripts. The dataset consists of ~30TBs of data and needs contains various climate models, each containing a few ensemble members, variables, and time-scales.

The calendar of each dataset may be different from others; something to be mindful about.

Enhancements required

Parallelization scheme for submitting jobs based on "models," SHOULD be implemented. This feature will be optional, as not all datasets, consist of various "models."

new dataset to be included

Could you please add a data processor tool for the complementary historical data set for bias corrected CanRCM4? The data is stored here and please refer to the paper in HESSD for reference. https://doi.org/10.5194/essd-12-629-2020
/project/6008034/Model_Output/181_WFDEI-GEM-CaPA_1979-2016

log path needs to be documented for Users

The title

`ERA5` subsetting does not cover the entire area

In some cases, the ERA5 subsetted output does not cover the entire area asked by the tool. Here is an example for the Bow River at Banff area:

Reported-by: Hongli Liu [email protected]

`rdrs` script may be lagged behind in processing small domains

There needs to be a until loop to assure the script finishes processing all the files. Sometimes for small domain, the script does not have a chance to copy all the files before it finishes.

Updating documentation

The documentation of the repository needs to be updated with relevant information, including READMEs and "docstrings."

`Ouranos ESPO-G6-R2` dataset workflow to be added

The title

`CONUS-II` future products lack leap days

Unlike CONUS-II historical runs, the future runs lack leap days. This needs to be taken care of for its processes.

`daymet` to be added to to the list of datasets

Problem

The daymet dataset could be added to the list of datasets.

Extra information

The link to the dataset website: https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=2129

no `--variable` means all available variables?

It would be convenient to avoid writing all --variable names, one can avoid writing all and expect to get the listed variables on the documentation page.

This can cause problem for the CONUS-II dataset that has ~200 variables.

Reported-by: Darri Eythorsson

Adding `ERA5 Land` dataset

The ERA5 Land dataset to be added to the tool.

Adding GCM downscaled data covering portions of AB-SK-BC-MT

The GCMs are not publicly available and the publication is under review. For now, the relevant workflows will be added here for later use.

Parallelizing based on `model`

Some datasets are comprised of several models, ensembles, and variables. Parallelizing based on models could be another parallelization scheme.

Issues with `CONUS I` and `CONUS II` list of variables

This is the table suggested by Zhenhua Li and Logan Fang in email exchanges to reflect the correct list of variables for CONUS I and CONUS II variables.

Variable Name	WRF-CONUSII Variable	Unit	IPCC abbreviation	Comments
surface pressure	PSFC	Pa	ps
specific humidity @2m	Q2	1	huss
air tempreature @2m	T2	k	tas
wind speed @10m	U10,V10	m/s	wspd	WIND=SQRT(U102+V102)
precipitation	PREC_ACC_NC	mm/hr		accumulated precipitation over one hour
short wave radiation	SWDOWN	W m-2	rsds
long wave radiation	GLW	W m-2	rlds

Reported-by: Logan Fang [email protected]

`SCDNA` dataset to be added

Problem

The SCDNA dataset to be added to the list of datasets.

Extra information

The relevant information could be found in the following: https://uofs-comphyd.github.io/group_datasets/SCDNA

Description of `RDRSv2.1` variables for the `40m` elevation is missing

Basically the title. These variable names end with a _09944 string. This need to be added to RDRS's documentation.

Introducing package dependencies

With this feature, the tool could be used in any "HPC" and the reliance on the specific configurations of each HPC will be minimized.

This is a pre-publication effort to assure the larger community can modify and use the main extract-dataset.sh script for various datasets of interest.

[Feature] Include data checking in other scripts

https://github.com/kasra-keshavarz/datatool/blob/963c65219ced0c1f377450decb48770ed5b4f322/scripts/eccc-rdrs/rdrs.sh#L248-L256

These lines limit the calls to subset RDRS to the period we actually have data for. This doesn't seem to happen in all scripts (e.g., Daymet) which results in a lot of error printouts about files not being found:

[..]
ncks: ERROR file /gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/temp/daymet/daymet_v4_daily_na_srad_1966.nc not found. It does not exist on the local filesystem, nor does it match remote filename patterns (e.g., http://foo or foo.bar.edu:file).
ncks: HINT file-not-found errors usually arise from filename typos, incorrect paths, missing files, or capricious gods. Please verify spelling and location of requested file. If the file resides on a High Performance Storage System (HPSS) accessible via the 'hsi' command, then add the --hpss option and re-try command.
ncks: ERROR file /gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/temp/daymet/daymet_v4_daily_na_tmax_1966.nc not found. It does not exist on the local filesystem, nor does it match remote filename patterns (e.g., http://foo or foo.bar.edu:file).
ncks: HINT file-not-found errors usually arise from filename typos, incorrect paths, missing files, or capricious gods. Please verify spelling and location of requested file. If the file resides on a High Performance Storage System (HPSS) accessible via the 'hsi' command, then add the --hpss option and re-try command.
[..]

`--account` needs to be provided as an interim solution

Since users from various "accounts" would want to use the script, the default "account" of the SLURM script needs to be exposed.

This will be re-adjusted in the future, since datatool will be flexible enough to work with various job submission scripts.

Making a new release `v0.4.1`

Two patches added with the recent commits:

The CONUS scripts now have correct paths to deal with the coordinate variables and relevant NCL script, since the update of the directory structure of the package as of v0.4.0,
The cache directory now has a new path, and is also compatible with $SLURM_TMPDIR. The --cache has always been an optional argument for users, but if they want to use the high throughput capability of Graham nodes' SSD storages, they can assign --cache='$SLURM_TMPDIR'. For now, users MUST be mindful of single quotation marks around $SLURM_TMPDIR variable. This will become a more convenient option for users soon to avoid confusion.
The conus_ii has been retested (no changes in code implemented), and works efficiently with the SSD cache path.

Therefore, a new release is required for the patches applied and tests implemented.

`WRF-CONUS II` production now has future runs from 2080-2100

The WRF-CONUS II production now has runs for the 2080-2100 period. The current script is not capable of handling this set of data. Furthermore, the data format is not in .tar anymore, and the outputs are simply nc files.

Reported by: Logan Fang

datatool creating a separate job submission without `-j` argument

Reported by @wknoben:

Random second thing, I'm having some odd outputs from datatool when run through external jobs (i.e., I make the job and run datatool inside it, it doesn't create its own job):

So this shows two different array job members, one has clean exist codes on everything, the other does not. Both logs seem almost identical (apart from the obvious differences in basin name and temporary folders). The one exception is that 498 encounters a bus error at the end:

/cm/local/apps/slurm/var/spool/job470014/slurm_script: line 15: 2421571 Bus error (core dumped) python 7b_rdrs_to_basins.py $SLURM_ARRAY_TASK_ID $SLURM_TMPDIR

I still need to check if the runs themselves actually failed (I do seem to have a lot of subset files) but I thought I'd share this just in case it's helpful


Lmod is automatically replacing "intel/2020.1.217" with "gcc/9.3.0".


Due to MODULEPATH changes, the following have been reloaded:
  1) mii/1.1.2

The following have been reloaded with a version change:
  1) StdEnv/2023 => StdEnv/2020          5) imkl/2023.2.0 => imkl/2020.1.217
  2) gcc/12.3 => gcc/9.3.0               6) libfabric/1.18.0 => libfabric/1.10.1
  3) gcccore/.12.3 => gcccore/.9.3.0     7) openmpi/4.1.5 => openmpi/4.0.3
  4) gentoo/2023 => gentoo/2020          8) ucx/1.14.1 => ucx/1.8.0

/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/scipy-stack/2022a/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.25.2
  warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
--- Now running basin 497. CAN_05ME003
    Basin coordinates:         [-101.08125      50.52708333 -100.45625      51.01458333]
    Flow obs unavailable:      []
    Flow obs times:            ['1953-08-01', '2022-11-01']
Running datatool command: /globalhome/wmk934/HPC/datatool/extract-dataset.sh --dataset="rdrs" --dataset-dir="/gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/temp/rdrs" --output-dir="/gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/camels-spat-data/basin_data/CAN_05ME003/forcing/raw/rdrs_day" --start-date="1953-08-01 00:00:00" --end-date="2022-11-01 00:00:00" --variable="RDRS_v2.1_A_PR0_SFC,RDRS_v2.1_P_TT_1.5m,RDRS_v2.1_P_FB_SFC,RDRS_v2.1_P_FI_SFC,RDRS_v2.1_P_HR_1.5m,RDRS_v2.1_P_HU_1.5m,RDRS_v2.1_P_P0_SFC,RDRS_v2.1_P_UUC_10m,RDRS_v2.1_P_VVC_10m,RDRS_v2.1_P_UVC_10m,RDRS_v2.1_P_GZ_SFC" --prefix="CAN_05ME003_subset_" --shape-file="/gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/camels-spat-data/basin_data/CAN_05ME003/shapefiles/lumped/CAN_05ME003_lumped.shp" --cache="/local/wmk934/470013"
CompletedProcess(args='/globalhome/wmk934/HPC/datatool/extract-dataset.sh --dataset="rdrs" --dataset-dir="/gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/temp/rdrs" --output-dir="/gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/camels-spat-data/basin_data/CAN_05ME003/forcing/raw/rdrs_day" --start-date="1953-08-01 00:00:00" --end-date="2022-11-01 00:00:00" --variable="RDRS_v2.1_A_PR0_SFC,RDRS_v2.1_P_TT_1.5m,RDRS_v2.1_P_FB_SFC,RDRS_v2.1_P_FI_SFC,RDRS_v2.1_P_HR_1.5m,RDRS_v2.1_P_HU_1.5m,RDRS_v2.1_P_P0_SFC,RDRS_v2.1_P_UUC_10m,RDRS_v2.1_P_VVC_10m,RDRS_v2.1_P_UVC_10m,RDRS_v2.1_P_GZ_SFC" --prefix="CAN_05ME003_subset_" --shape-file="/gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/camels-spat-data/basin_data/CAN_05ME003/shapefiles/lumped/CAN_05ME003_lumped.shp" --cache="/local/wmk934/470013"', returncode=0, stdout='(2024-05-26 02:53:39) rdrs.sh: processing ECCC RDRSv2.1...\n(2024-05-26 02:53:39) rdrs.sh: creating output directory under /gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/camels-spat-data/basin_data/CAN_05ME003/forcing/raw/rdrs_day\n(2024-05-26 02:53:39) rdrs.sh: creating cache directory under /local/wmk934/470013\n(2024-05-26 20:00:37) rdrs.sh: deleting temporary files from /local/wmk934/470013\nsending incremental file list\ndeleting 2018/2018123012.nc\ndeleting 2018/2018122912.nc\ndeleting 2018/2018122812.nc(2024-05-26 20:00:40) rdrs.sh: temporary files from /local/wmk934/470013 are removed\n(2024-05-26 20:00:40) rdrs.sh: results are produced under /gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/camels-spat-data/basin_data/CAN_05ME003/forcing/raw/rdrs_day\n', stderr="-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: gdal/3.4.1 (required by: geo-stack/2022a), proj/9.0.0 (required by: libgeotiff/1.7.1, libspatialite-proj9/5.0.1, geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: gdal/3.4.1 (required by: geo-stack/2022a), proj/9.0.0 (required by: libgeotiff/1.7.1, libspatialite-proj9/5.0.1, geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: eccodes/2.25.0 (required by: geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: eccodes/2.25.0 (required by: geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: eccodes/2.25.0 (required by: geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: eccodes/2.25.0 (required by: geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: gdal/3.4.1 (required by: geo-stack/2022a), netcdf/4.7.4 (required by: eccodes/2.25.0, geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: gdal/3.4.1 (required by: geo-stack/2022a), netcdf/4.7.4 (required by: eccodes/2.25.0, geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: eccodes/2.25.0 (required by: geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: eccodes/2.25.0 (required by: geo-stack/2022a)\n-------------------------------------------------------------------------------\nmkdir: cannot create directory ‘/globalhome/wmk934/HPC/empty_dir’: File exists\nrm: cannot remove '/local/wmk934/470013': Permission denied\n")


Lmod is automatically replacing "intel/2020.1.217" with "gcc/9.3.0".


Due to MODULEPATH changes, the following have been reloaded:
  1) mii/1.1.2

The following have been reloaded with a version change:
  1) StdEnv/2023 => StdEnv/2020          5) imkl/2023.2.0 => imkl/2020.1.217
  2) gcc/12.3 => gcc/9.3.0               6) libfabric/1.18.0 => libfabric/1.10.1
  3) gcccore/.12.3 => gcccore/.9.3.0     7) openmpi/4.1.5 => openmpi/4.0.3
  4) gentoo/2023 => gentoo/2020          8) ucx/1.14.1 => ucx/1.8.0

/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/scipy-stack/2022a/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.25.2
  warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
--- Now running basin 498. CAN_05ME005
    Basin coordinates:         [-101.30958333   50.78041667 -101.05375      50.93041667]
    Flow obs unavailable:      []
    Flow obs times:            ['1959-03-01', '2022-11-01']
Running datatool command: /globalhome/wmk934/HPC/datatool/extract-dataset.sh --dataset="rdrs" --dataset-dir="/gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/temp/rdrs" --output-dir="/gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/camels-spat-data/basin_data/CAN_05ME005/forcing/raw/rdrs_day" --start-date="1959-03-01 00:00:00" --end-date="2022-11-01 00:00:00" --variable="RDRS_v2.1_A_PR0_SFC,RDRS_v2.1_P_TT_1.5m,RDRS_v2.1_P_FB_SFC,RDRS_v2.1_P_FI_SFC,RDRS_v2.1_P_HR_1.5m,RDRS_v2.1_P_HU_1.5m,RDRS_v2.1_P_P0_SFC,RDRS_v2.1_P_UUC_10m,RDRS_v2.1_P_VVC_10m,RDRS_v2.1_P_UVC_10m,RDRS_v2.1_P_GZ_SFC" --prefix="CAN_05ME005_subset_" --shape-file="/gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/camels-spat-data/basin_data/CAN_05ME005/shapefiles/lumped/CAN_05ME005_lumped.shp" --cache="/local/wmk934/470014"
CompletedProcess(args='/globalhome/wmk934/HPC/datatool/extract-dataset.sh --dataset="rdrs" --dataset-dir="/gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/temp/rdrs" --output-dir="/gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/camels-spat-data/basin_data/CAN_05ME005/forcing/raw/rdrs_day" --start-date="1959-03-01 00:00:00" --end-date="2022-11-01 00:00:00" --variable="RDRS_v2.1_A_PR0_SFC,RDRS_v2.1_P_TT_1.5m,RDRS_v2.1_P_FB_SFC,RDRS_v2.1_P_FI_SFC,RDRS_v2.1_P_HR_1.5m,RDRS_v2.1_P_HU_1.5m,RDRS_v2.1_P_P0_SFC,RDRS_v2.1_P_UUC_10m,RDRS_v2.1_P_VVC_10m,RDRS_v2.1_P_UVC_10m,RDRS_v2.1_P_GZ_SFC" --prefix="CAN_05ME005_subset_" --shape-file="/gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/camels-spat-data/basin_data/CAN_05ME005/shapefiles/lumped/CAN_05ME005_lumped.shp" --cache="/local/wmk934/470014"', returncode=0, stdout='(2024-05-26 02:53:41) rdrs.sh: processing ECCC RDRSv2.1...\n(2024-05-26 02:53:41) rdrs.sh: creating output directory under /gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/camels-spat-data/basin_data/CAN_05ME005/forcing/raw/rdrs_day\n(2024-05-26 02:53:41) rdrs.sh: creating cache directory under /local/wmk934/470014\n(2024-05-26 20:00:37) rdrs.sh: deleting temporary files from /local/wmk934/470014\nsending incremental file list\ndeleting 2018/2018123012.nc\ndeleting 2018/2018122912.nc\ndeleting 2018/2018122812.nc
(2024-05-26 20:00:45) rdrs.sh: temporary files from /local/wmk934/470014 are removed\n(2024-05-26 20:00:45) rdrs.sh: results are produced under /gpfs/tp/gwf/gwf_cmt/wknoben/camels_spat/camels-spat-data/basin_data/CAN_05ME005/forcing/raw/rdrs_day\n', stderr="-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: gdal/3.4.1 (required by: geo-stack/2022a), proj/9.0.0 (required by: libgeotiff/1.7.1, libspatialite-proj9/5.0.1, geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: gdal/3.4.1 (required by: geo-stack/2022a), proj/9.0.0 (required by: libgeotiff/1.7.1, libspatialite-proj9/5.0.1, geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: eccodes/2.25.0 (required by: geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: eccodes/2.25.0 (required by: geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: eccodes/2.25.0 (required by: geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: eccodes/2.25.0 (required by: geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: gdal/3.4.1 (required by: geo-stack/2022a), netcdf/4.7.4 (required by: eccodes/2.25.0, geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: gdal/3.4.1 (required by: geo-stack/2022a), netcdf/4.7.4 (required by: eccodes/2.25.0, geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: eccodes/2.25.0 (required by: geo-stack/2022a)\n-------------------------------------------------------------------------------\n-------------------------------------------------------------------------------\nThe following dependent module(s) are not currently loaded: eccodes/2.25.0 (required by: geo-stack/2022a)\n-------------------------------------------------------------------------------\nmkdir: cannot create directory ‘/globalhome/wmk934/HPC/empty_dir’: File exists\nrm: cannot remove '/local/wmk934/470014': Permission denied\n")
/cm/local/apps/slurm/var/spool/job470014/slurm_script: line 15: 2421571 Bus error               (core dumped) python 7b_rdrs_to_basins.py $SLURM_ARRAY_TASK_ID $SLURM_TMPDIR

Issue description

The project could be briefly described for the MESH modelling community to communicate the features of the tool as suggested by Ala Bahrami.

ch-earth / datatool Goto Github PK

datatool's Introduction

Description

Available Datasets

General Example

Logs

New Datasets

Support

License

Footnotes

datatool's People

Contributors

Stargazers

Watchers

Forkers

datatool's Issues

Problem

Extra information

Issue description

Dataset

Issue description

problem statement

Problem statement

Problem

Enhancements required

Problem

Extra information

Problem

Extra information

Issue description

Recommend Projects

Recommend Topics

Recommend Org