"Agnostic Orchestrator" is a simple bash
script aiming to simplify
executing various processes involved in preparing meteorological and
geospatial data for setting up various hydrological models following the
concept of "model-agnostic framework". For a complete description of such
a framework, see Knoben et al. 2022.
The "Agnostic Orchestrator" is able to submit various dependant and
independant jobs to the SLURM job scheduler that is widely used on
most High-performance Computing facilities (HPCs). Independant jobs are
submitted without following any hierarchy, while dependant jobs are
submitted following an order of execution that prepares necessary
meteorological and geospatial datasets for setting up hydrological models.
The order of executation follows the dependency
capability of the SLURM
scheduler to submit jobs based on a hierarchy given.
The script uses a JSON
configuration file to specify the:
- executables,
- relevant options and arguments of the provided executables,
- and hierarchy of job submissions to the scheduler.
The script is highly optimized to be used on Digital Research Alliance of Canada (DRA)'s Graham HPC. The executables that are commonly used with this script is also highly optimized for the mentioned computing system.
To use the "Agnostic Orchestrator", please remember that you MUST have access to Graham cluster with Digital Research Alliance of Canada (DRA) 1.
To access the script, simply access this online repository locally:
foo@bar:~$ git clone https://github.com/kasra-keshavarz/agnostic-orchestrator.git # clone the repository
A sample of such a configuration file is in the following:
{
"exec": {
"met": "/path/to/extract-dataset.sh",
"gis": "/path/to/extract-gis.sh",
"remap": "easymore cli"
},
"args": {
"met": [{
"dataset": "RDRS",
"dataset-dir": "/path/to/RDRS/",
"variable": [
"RDRS_v2.1_P_P0_SFC",
],
"output-dir":"/path/to/outputs",
"start-date":"1980-01-01T13:00:00",
"end-date":"1980-12-31T12:00:00",
"shape-file": "/path/to/catchments.shp",
"_flags": [
"submit-job",
"parsable"
]
}],
"gis": [{
"dataset": "landsat",
"dataset-dir": "/path/to/Landsat/",
"variable":"land-cover",
"output-dir":"/path/to/landsat",
"shape-file": "/path/to/catchments.shp",
"print-geotiff": "true",
"stat": [
"frac",
"coords"
],
"_flags": [
"include-na",
"submit-job",
"parsable"
]
}],
"remap": [{
"case-name":"ab_models",
"cache": "/path/to/temp_dir/",
"shapefile": "/path/to/catchments.shp",
"source-nc": "/path/to/*.nc*",
"variable": [
"RDRS_v2.1_P_P0_SFC",
],
"output-dir": "/path/to/outputs/remapped/",
"_flags": [
"submit-job"
]
}]
},
"order": {
"met": 1,
"gis": -1,
"remap": 2
}
}
As is shown in the JSON
block above:
- First, the executables are listed and their paths are entered,
- Second, the arguments of the exetubales are detailed,
- And finally, the order of job submission to the SLURM scheduler is
given, with
order
values of-1
indicating independant jobs.
The "Agnostic Orchestrator" can be run as following:
foo@bar:~$ ./agnostic-orch.sh agnostic-conf.json
An example to prepare meteorological and geospatial datasets to prepare
MESH
hydrological/land-surface model is given in the example
directory of the current repository.
Please open a new ticket on the Issues tab of the current repository in case of any problem.
Agnostic Orchestrator Workflow
Copyright (C) 2023, University of Calgary
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.
Footnotes
-
For access, please contact Kasra Keshavarz at "kasra.keshavarz1 AT Ucalgary DOT ca" โฉ