Windows, IE 11 (and also previous versions).

Promise is undefined, need polyfill.

Will ask once GetFeatureInfo request is ready for their review

SMS Notification System

needs

• Support people subscribing a phone number to a flood gauge to receive updates when a gauge enters a flood stage or is predicted to. This first subscription step will be triggered from the front-end via HTTP request.
• The data will be scraped/updated periodically from National Weather Service in the data pipeline (#36).
  • Data pipeline will trigger this notification system, which should send out messages to every subscribed number telling them what’s up. Messages should be able to contain a URL with more info.
• Twilio and Plivo look like solid third party services handling the message sending part. AWS SNS service can kind of do this, but is really limited and missing a lot of features.
• Needs to handle user flow (“You’ve subscribed to updates from TXFLOOD… send STOP to cancel”).
• Both Twilio and Plivo handle this automatically.
• Should not spam or blow up anyone's phone.

status

• research available third-party SMS systems
  • procure services (3-4 week lag time)
• design
• implementation
• deployment

A continuous map layer of precipitation (measured from doppler radar). Preferably Animated.

status

• evaluate available services
  • weather underground
    • only static maps available are available in a useable format animated maps are flash video which doesn't work well for a lot of reasons
  • Iowa state mesonet WMS
    • we have a few concerns based on previous experience using as a weather layer for GEMSS2
      • it is sometimes slow or unavailable
      • it probably will not scale to ~10000's of users
    • if we use it, we will probably need to set up and maintain a cached version of the service, which is possible but not trivial
  • aerisweather.com
    • animated maps are available as a tile service which is easy to work with
    • as a platform, Aeris provides many other layers that we might want to integrate in the future
    • looks like this is the winner
• procure Aeris subscription #31
• integrate into map viewer

Real-time road closure information maintained by TxDOT for DriveTexas

source

• Looks to be cached and served out for the Drive Texas app by a service called MapLarge that makes it difficult to get to the raw data

status

• @mpavon in communication with TxDOT about allowing us access to the dataset. Discussions have been slow going due to recent re-org at TxDOT.

Lake conditions

source

• preliminary data available at http://waterdatafortexas.org/reservoirs/recent-conditions.json
• there is a better real-time service is being currently worked on by WDFT staff that will provide height above conservation pool elevation and height to the top of the dam

challenges

• flood stage is not a good proxy for flood danger
  • there are a lot of other factors that determine likelihood of flood releases and these are different for each reservoir
  • rating curves stop at conservation pool elevation, so we don't have a good estimate of flood pool volume

shouldn't be too hard, but there might be weird interactions with external map layers if they don't support it

This depends on #71 so we can figure out what we need to get from where.

Need proper attribution for the Texas Google Imagery base layer. I've emailed tnrisdatasupport asking for the proper language.

This morning the nws_ahps_gagues_texas auto-sync table was completely empty. I clicked the "Sync now" link in the CartoDB web UI and it successfully grabbed the data.
Obviously a big part of this project is going to be managing these types of sync problems.

Would be good to order the point rendering by flood severity, with most severe dots rendering on top of less sever.

If the async call in the AnimatedWeaterLayer's update() method has not finished by the time addTo(map) is called, an error will be thrown during this line: this.layers[i++].setOpacity(0.8) (because layers[0] will be undefined)

Where possible, use 2x resolution tiles for high DPI screens (retina devices, 4k monitors, fancy phones and tablets). Maybe we shouldn't do this for mobile devices, where bandwidth matters.

CartoDB layers directly support the common '@2x' pattern. Basemaps are tricky and only some have will support high resolutions. Strictly aerial layers can be tricked into rendering one z level deeper and that'll work fine but probably not worth the complexity. Non-photo rendered maps generally look weird when trying to do this.

We need to purchase an Aeris subscription. Until then advisory layers and api queries (like for flood alerts) won't be available.

will work better for any size of screen / device

A service that pushes updates to CartoDB when new data is available.

needs

• data sources will be periodically scraped and processed via data pipeline (see: #36)
• when updates are ready, this service will
  • determine what updates need to be made
  • batch them and update tables on CartoDB via the SQL API
• see this blog post: http://blog.cartodb.com/election-maps-with-the-power-and-beauty-of-cartodb for the general approach
• we are currently using CartoDB's sync feature for the prototype, but it has some limitations:
  • sync can be triggered, at best, every 15 minutes
  • does not fail gracefully
    • does not have any error handling or logs, so not possible to determine why it failed
    • cannot notify us when it fails
    • the dataset table ends up blank when this happens, meaning the map layer will be blank
    • usually requires a "hard reset" by either generating a duplicate version of the dataset/visualization or contacting by CartoDB support

status

• design
• implementation
• deployment

After displaying the AnimatedWeatherMaps everything else becomes a bit sluggish.

Trying to nail down exactly where the memory/perf leak is, but I think it has to do with the browser hanging on to 20 layers worth of images.

Will need to experiment with where exactly that goes. It could go in with the map selector or somewhere on top of the map itself.

This issue is to track high level data pipeline and service architecture discussion

data pipeline needs

• data will be collected from different sources (such as NWS, USGS, Texas Mesonet, MesoWest, and many more)
• pipeline needs to take data through Extract, Transform, Load (ETL) and allow for additional processing steps
• will load data into some persistent, queryable data store (probably either DynamoDB or postgresql hosted via RDS)
• modular:
  • each data source works differently, so our pipeline should accommodate that
    • have its own ETL process and schedule as appropriate
    • ideally we would be able to use any script, CLI tool, etc so that we don't need to reinvent any wheels
  • we should try to keep it simple as possible so we can plug in new people and distribute workload
• fault-tolerant:
  • if a service is unavailable for any reason, then retry after a short timeout
  • if a service is down for a while, email notifications should be sent to us (small number of people) to take a look at it
  • always keep last best data available in the viewer
• be kind:
  • must not overwhelm (Denial of Service) any data sources
  • if a ton of API calls or HTTP requests need to be made, then orchestrate and rate-limit those requests
• allow for aggregation of multiple data sources for display
  • example: combine NWS flood gauges with USGS streamflow gauges for display on the map
    • deduplicate any gauges that exist in both sources with preference to NWS data
  • aggregation exists outside of the ETL processes for the source data
• able to trigger external events at any step (when data is ready)
  • example: trigger SMS notification service for gauge subscriptions when a gauge is predicted to flood
  • example: push-to-cartodb service that sync's ready-for-display output with CartoDB tables
  • these external services should only be loosely coupled to the pipeline
  • exact trigger mechanism to be determined, but an example would be via HTTP callback or message queue

data service needs

• service just needs to have a simple API to support charts and graphs on front-end
  • most likely a small wrapper around the datastore
  • output as JSON
  • queries will answer questions like
    • "give me all the data at gauge from '2016-03-17' to now"
    • "give me all the streamflow gauges within bbox"
      • this can be done with simple ranges and an x and y for gauges, does not need to be a full-blown geospatial engine
      • more complete geospatial queries like distance would be nice to have, but not critical
• fault-tolerant: the data service is critical
• scalable
  • for sake of planning, assume we will need to be able to handle ~10000 users at peak
  • peaks will be during flood events, with low traffic in between

considerations

• realtime is important
  • data provided need to be as timely as possible
  • we'll need to resort to polling for initial pipeline Extract step for but the following steps should be able to trigger each other via notifications, events or message queue
• most data will timeseries data, so timeseries should be well supported
  • need to be able to handle timeseries queries
  • use good practices for timeseries data
    • store all timestamps as UTC, convert to UTC at the very beginning of pipeline
    • will be converted to local timezones at last possible step, for presentation to users (so intermediate steps don't need to worry about it)
    • if there is no native datetime type, use ISO 8601 strings
• we don't have an ops team to keep things afloat, so we need to lean heavily on managed services
• we have access to virtually any of the AWS services
• additional third-party services are possible but procurement is tricky and will take 3-4 weeks

IE has no support for Object.assign: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/assign

We should make sure to always use the object-assign package.

This issue splits out discussion of what others have called the data service.

To rephrase what others have said (corrections welcome), the data service is a simple HTTP JSON API which the front end can use to get data about gauges, e.g.

• "give me all the data at gauge from '2016-03-17' to now"
• "give me all the streamflow gauges within bbox"

I believe these two queries imply certain capabilities

• Range queries on timestamps
• Bounding box queries, i.e. combining range queries on x and y coordinates

Data service should handle heavy, bursty read loads, and it should not go down very much. That defines these dependencies, partly concerning where the API itself runs but mostly the datastore:

• Load testing to explore and then demonstrate these properties #3
• Either provision capacity in advance, or set up some kind of autoscaling to manage capacity

To be available by May 1.

Needs to have a few live (near-realtime) data layers:

• NOAA NWS gauges w/predictions and flood stages: #37
• USGS NWIS streamflow gauges: #42
• Precipitation (Doppler): #39, #31
• NWS Watches & Warnings: #40
• Low-water crossing closures (from atxfloods.com): #41
• WaterDataForTexas reservoir current conditions: #38

Texas Mesonet

• This is a the new network of weather stations and rain gauges that is being developed and deployed by TWDB
• Will only contain precipitation data (inches of rainfall), windspeed, temperature data rather than streamflow

status

• web services still being built there is an internal dev version with empty values, but nothing public-facing yet
• TWDB working hard on the land owner agreements that need to be there in order for instruments to be deployed (as of mid-March)
• ETA: early April, if we are lucky

I'm making this issue for discussion of the interface that will be provided from DynamoDB to Data Service.

This could be worked out starting from higher-level considerations like what kinds of data will be stored, and what kinds of access patterns are expected. Or it could be implied by some code. Or the schema could just be given directly. Whatever. It just needs to be worked out somehow.

Dependencies of this issue include both the Data Service API itself and also the pipeline code which puts the data for Data Service into DynamoDB, and therefore needs to know about things like hash keys and range keys.

Slightly hard to understand, but we might to display the Aeris logo on the page.

http://www.aerisweather.com/support/docs/credit/ says:

While some of our products and services can be used commercially without requiring attribution, most products including the AerisWeather API and Aeris Maps require proper attribution by using any one or a combination of the allowed AerisWeather marks as described below.

This issue tracks progress on implementing gauges for the National Weather Service (NWS) Advanced Hydrologic Prediction Service (AHPS)

source

• There is a “map points” endpoint that returns a some JSON with each gauge, what are hitting for our half-baked prototype, and that is what actually powers the web map on the AHPS site.
  • example: curl 'http://water.weather.gov/ahps/get_map_points.php' --data 'key=tx&fcst_type=obs&percent=50&current_type=all&populate_viewport=1&timeframe=0' —compressed
• A better map service is available at https://idpgis.ncep.noaa.gov/arcgis/rest/services/NWS_Observations/ahps_riv_gauges/MapServer - we should probably use this instead
• For each gauge, there is an XML endpoint and separate RSS feeds for observed, forecast, and alerts.
• The most complete version of the data is XML endpoint for the hydrograph. It contains all of the important info in one place and is way cleaner to parse than the RSS feeds.
  • For example, the hydrography XML endpoint for the gauge with code “TDDT2” is: http://water.weather.gov/ahps2/hydrograph_to_xml.php?gage=tddt2&output=xml
• In the XML, there are a few elements:
  • <disclaimers> are the XML version of “for entertainment purposes only” messages. We don’t need to worry about these
  • <sigstages> contains the set of Flood Stages defined for the gauge. These are important, we need these to associate observed and forecast values to flood stages.
  • <sigflows> is the analog with stages defined by streamflow, but seem to be incorrect sometimes. Probably should ignore
  • <zerodatum> TL;DR is “its kind of complicated and we don’t need to worry about it for this application”. Longer version is that it defines in a semi-precise way what “10 ft” of stage means. It references a more or less consistent model of the earth, so “235.49 above Mean Sea Level” means that a stage of 0ft is 235.49 ft above MSL and a flood stage of 5ft is 240.49 ft above MSL. It gets more complicated than that because "Sea Level" is a weird construct, but I’ll stop there.
  • <rating> is the rating curve for the gauge. We don’t need to worry about this, because the values will be contain both Stage and Streamflow with the conversion automatically applied.
  • <alt_rating> an alternate version of the rating curve? Again, feel free to ignore.
  • <observed> Observed values that were measured by the gauge. These have already happened.
  • <forecast> The modeled forecast of what is predicted to happen. These will not always be available - NWS only runs forecasts when they have determined that there will be some flood risk.
  • So to recap, we only need the stage and streamflow data from <observed> and <forecast> (when available) and stage needs to be correlated it with to assign a flood stage category.

• need to get advice from TWDB Legal department on what language and disclaimers need to be on the site
• specifically, features like real-time SMS notifications raise questions with regard to liability

something like a spinner or progress bar to indicate when a layer is not quite ready to be added to the map yet

A map layer containing active Watches and Warnings from the National Weather Service

source

• available through Aeris weather, which updates every 2-3 minutes
• original source of data is NWS: http://www.weather.gov/
• have a rough pass at a five-minute updating geojson layer at http://labs.tnris.org/alerts.geojson

status

• waiting on procurement of Aeris subscription #31

absolute garbage at the moment, mostly due to a combination of S3 being slow and a single large scripts.js file that includes a bunch of vendor'd code:

Some time before launch this needs to be fixed with some combination of:

• deploying on cloudfront
• code splitting
• tree shaking (if we wanna get real fance)
• displaying something other than blank screen while the app loads

A map layer containing active Watches and Warnings from the National Weather Service

source

• available through Aeris weather, which updates every 2-3 minutes. See the "Severe" tab on http://www.aerisweather.com/support/docs/aeris-maps/reference/map-layers/
• There is even a layer just for "Alerts (Flood Related)"
• original source of data is NWS: http://www.weather.gov/
• have a rough pass at a five-minute updating geojson layer at http://labs.tnris.org/alerts.geojson

status

• waiting on procurement of Aeris subscription #31

Aggregated set of weather stations

source

• website is http://mesowest.org/
• don’t have a good handle on this one – just heard about it recently and I haven’t really looked at it

Map layer showing forecast precipitation

source

• Aeris provides a layer with some rain and radar forecasts that we can use for MVP
  • these layers require subscription, so waiting on procurement (#31)
  • integrate into map viewer
• There are some more official forecasts provided by NOAA and NWS but they will require more work than we have time for.

for NWS Flood Gauge layer (#37) and Lake Conditions (#47)

• map layer to represent WDFT lake conditions, based on flood pool elevation #38
• Yujuin's recommendation is equally divide the distance from conservation pool top to of them dam into 5 levels: 0-20%, 20-40%, 40-60%, 60-80%, and over 80%

status

• waiting on flood pool information to be available via WDFT

It always causes a page refresh.

CartoDB layers should re-request their layerids (via post to https://tnris.cartodb.com/api/v1/map/) on a timer. Probably a 2 to 4 hour timer would work to ensure that the tile layers don't start to error out after long periods of user inactivity.

Streamflow data from the USGS National Water Information System

source

• Huge network of gauges run by USGS
• Is a super-set of the NWS AHPS gauges (#37) but USGS source does not contain info about flood stage or predictions, only observed values
• Well documented at: http://waterservices.usgs.gov/rest/IV-Service.html
• Ulmo can be used to download & parse the data: https://github.com/ulmo-dev/ulmo

status

• waiting on #36 to determine how to implement

i got this email from cartodb. 5 so far since 350am.

The animated frames switch before all tiles load, causing a weird patch-work for the first few iterations before each layer has finished loading. Doesn't look good.

A simple-ish solution would be to wait for each timestamp layer to finish loading before moving on. Should also have a few second timeout so that a dropped tile request doesn't hang up the whole app

To implement #46, we need an API token tied to the TNRIS CartoDB account.

For security, we should create a limited user only for use in this project. Ideally, I'd like this user (and likewise, this user's token) to only have UPDATE/INSERT access to the table(s) used in this project. We can use an admin user to create & delete table(s) as needed.

Multiuser access is only available to Enterprise CartoDB accounts, so this may not be possible on the current TNRIS account. The alternative just means we have a singleuser account, with one god-token that allows R/W access to all tables. That technically would still work, but we might want to avoid distributing said god-token across production servers.

Road closure data from https://www.atxfloods.com/

source

• data published in XML format and pretty easy to parse (see: https://github.com/TNRIS/atx-low-water-crossings)
• coverage is limited to only a few counties in central Texas
• might end up as a backup data source for the Road Closures layer if we can't get TxDOT road closures off the ground

set up deployment for front-end code -> S3 with proper DNS records

Windows 10, Chrome

Clicking the fullscreen button on the leads to a fullscreen black box (and also showing the browser's "Press ESC to exit fullscreen" message). No errors appear to be thrown.

If the dev console is open, the fullscreen button works as expected.

Fullscreen button works as expected in Windows 10 Firefox.

Come up with a way of doing stress testing, to ensure high availability under heavy load.

Possible options:

• beeswithmachineguns
• goad