Cached FIM HydroTable - Lazy Loading about hydrovis HOT 3 CLOSED

Updates for documentation:

• Spring 2023: Initial results were successful in loading flood stacks for ~100 HUC8s into Redshift. Inundation was queried at sub-second timing for this subset, however, loading the data took several days running on an EC2. Estimates suggest 70+ days for a single EC2 machine to load the entire FIM Dataset library.
• Spring 2023: We then contacted the FIM Dev team to see if they could easily add in the vector storage into their processes. After some back and forth with Dr. Tarboton, Dr Yan Liu wrote a helpful script that calculates the hydrotable geometries after the fact, but it was concluded that writing the vectorization into TauDEM was not a good strategy.
• Summer 2023: We are proceeding with implementation of the Lazy Loading approach, where we'll cache flooded hydrotable entries in Redshift, and only run FIM in real-time for unchanged features/discharge combinations.

from hydrovis.

Summer 2023 Tasks:

• Implement flood stack / geocurve approach with RAS2FIM (Brad and Corey)
• Implement lazy loading for HAND FIM (Tyler)

from hydrovis.

We're finally moving forward with implementing this approach for HAND FIM. Here's the plan:

• Setup Redshift database (Tyler)
• Setup Dev pipeline in TI (Shawn)
• Cached FIM Development:
  • FIM Data Prep Lambda Function - Check Redshift table and remove applicable reaches from reaches for which to run HAND
  • FIM HUC Processing Lambda Function - Write geometries to Redshift table instead of RDS
  • Postprocess SQL Lambda Function - Query Redshift table instead of RDS for FIM queries. Still write to RDS publish schema (need to experiment to see if this should happen directly from LAMBDA or a PostgreSQL Foreign Data Table)
  • Setup record count, performance, and cost metrics. Run test events / generate reports / estimates to report back to the group.
• Probabilistic FIM Development
  • Initialize Pipeline Lamba Function - MRF Mem 2-6 Pipelines
  • Generate prototype Maps (Urgent - Tyler)
  • Ingest Lambda Function - Ingest ensemble members to Redshift instead of RDS
  • Postprocess SQL - Probabilistic FIM products

from hydrovis.