noaa-owp / hydrovis Goto Github PK

• #258
• Add CSV to DB
• Export dump

• Receive scripts and data from coastal team
• Test coastal scripts as received in AWS
• Modify workflow for AWS optimization/implementation
• Create example services for review
• Commit all relevant changes to GitHub repositories
• Implement into pipeline
• Updated SD creation script for mosaic dataset
• Test workflow with NWM 3.0 test feed on NOMADs
• Implement SCHISM FIM when NWM 3.0 goes live

• Update Services admin table
• Update jupyter raster functions
• Create python file
• Create SQL file
• Create Pro Project
• Testing/QCQA

• ana_high_flow_magnitude
• ana_high_flow_magnitude_hi
• ana_high_flow_magnitude_prvi
• ana_streamflow
• ana_streamflow_hi
• ana_streamflow_prvi
• srf_high_water_arrival_time
• srf_high_water_arrival_time_hi
• srf_high_water_arrival_time_prvi
• srf_high_water_probability
• srf_max_high_flow_magnitude
• srf_max_high_flow_magnitude_hi
• srf_max_high_flow_magnitude_prvi
• srf_peak_flow_arrival_time
• srf_peak_flow_arrival_time_hi
• srf_peak_flow_arrival_time_prvi
• srf_rapid_onset_flooding
• srf_rapid_onset_flooding_probability
• srf_rate_of_change

• AnA Streamflow
• AnA 72 Hour Accumulated Precipitation
• SRF Accumulated Precipitation
• MRF Accumulated Precipitation
• SRF Peak Flow Arrival Time
• MRF Peak Flow Arrival Time

We need a basic Redshift database to hold the anomaly percentile and ensemble tables. First thing to do is load the percentile data and test out running anomaly query from within postgis against redshift foreign table to see how viable this is long-term. Shouldn't take more than a half-day or so.

A redshift database can also provide the framework for a hosted archive version of each service, which would store service data for the past X days. Hosting this data via ESRI map service would provide a low-effort API for WPOD to access archived data through 1) a geoprocessing tool (currently being developed by Don) and/or a time-slider enabled web map.

• AnA 72 Hour Accumulated Precipitation
• AnA 72 Hour Accumulated Precipitation - Hawaii
• AnA 72 Hour Accumulated Precipitation - Puerto Rico
• SRF Accumulated Precipitation
• SRF Accumulated Precipitation - Hawaii
• SRF Accumulated Precipitation - Puerto Rico

Update the repo directory structure to be based on application, not on AWS service

Let's try migrating the EC2 deploy script that creates service definition files into a hosted geoprocessing service on the server. This service would use the mapx files from the repo to create SD files that the publish lambda function can use the publish.

• Draft geoprocessing service and run in ArcPro
• Publish to TI (with Bob's help)
• Figure out plan to use custom Python environment on AGS (waiting for input from Justin)
• PR with IAM role tweak
• Establish plan for publishing to other environments with Bob

postprocess_sql is failing on the fimpact building footprints summary step occasionally in TI. I'm looking into this and running some tests.

• Update Services admin table
• Update jupyter raster functions
• Create python file
• Create SQL file (service_metadata table)
• Create Pro Project
• Testing/QCQA

We will need to reproduce the AEP FIM data using the latest FIM4 version

• Recreate data using step functions
• Dump new aep_fim schema
• Push new dump file to S3

• Update Services admin table
• Update jupyter raster functions
• Create python file
• Create SQL file
• Create Pro Project
• Testing/QCQA

• Add new static dataset to the EGIS DB
• Save EGIS DB schema as a dump file
• Add new pertinent rating curves to VIZ DB
• Save VIZ DB schema as a dump file
• Move all dump files to S3

• ana_inundation
• ana_inundation_hi
• ana_inundation_prvi
• srf_max_inundation
• srf_max_inundation_hi
• srf_max_inundation_prvi

• Add stage-based CatFIM to the DB
• Add flow-based CatFIM to the DB
• Update stage-based CatFIM mapx to use intervals
• Dump EGIS DB
• Update dump file in S3

• Update Services admin table
• Update jupyter raster functions
• Create python file
• Create SQL file
• Create Pro Project
• Testing/QCQA

• Update Services admin table
• Update jupyter raster functions
• Create python file
• Create SQL file
• Create Pro Project
• Testing/QCQA

• NBM Accumulated Precip
• GFS Accumulated Precip
• NBM High Water Arrival Time
• GFS High Water Arrival Time
• NBM Peak Flow Arrival Time
• GFS Peak Flow Arrival Time
• NBM Max High Flow Magnitude
• GFS Max High Flow Magnitude
• NBM Rapid Onset Flooding
• GFS Rapid Onset Flooding
• GFS High Water Probability
• GFS Rapid Onset Flooding Probability
• NBM Max Inundation
• GFS Max Inundation

• Rerun catchment workflow
• Save EGIS DB outputs as a dump file
• Add dump file to S3

• Setup Route 53 with EGIS DB
• Setup Route 53 with Viz Fileshare
• Update EGIS DB part of RDS Bastion to create hydrovis DB and users as needed

• Update initialize pipeline lambda to handle an eventbridge event
• Update admin services table
• Save service_metadata.ipynb
• Update viz classes to handle an eventbridge event
• Update ingest to copy para data to S3 for saving/using data

• Update Services admin table
• Update jupyter raster functions
• Create python file
• Create SQL file
• Create Pro Project
• Testing/QCQA