taylorterry3 / covid_dc Goto Github PK

A friend sent this to me as a link, so I need to document where it came from and what I did to it in the README

A few new school names have shown up in the data set, so they need to be added to the decode dict.

About 450 letters from now I'll need to repeat the process in #14 so we don't lose data as letters fall off the page they're scraped from.

Add a side table of ward, address, DCPS codes, and other info for each school. This will also require creating a canonical short name for each school.

Right now this system scrapes everything every time, which will stop working soon because the source page only shows the most recent 500 letters (see https://dcpsreopenstrong.com/health/response/notifications/). I had planned to switch to reading the PDFs and keep the old ones in the repo, which would be slow but stateless. The PDFs have been buggy and occasionally malformed, though, so I need a plan B.

When the year rolls over there will be a need to fix things that assume that the dates being parsed are in the current year. This may just amount to versioning the data through December and starting a fresh round in January.

Similar to #14 , the PDF's are about to spill off into the dustbin of history and need to be archived.

Right now this is just happening in the notebook in two lines, but it needs some null handing added and should go in a function.

Because some of the one-shots find and replace data within the scraped data they will need to be versioned along with the archived data so they aren't looking for things that aren't there. Another option would be to handle the error.

existing functions could use some tests

I've been pretty inconsistent with adding type annotations outside of function signatures, so at some point I need to fix that.

Add elementary/middle/high/other field to dataset. This will require a decode dict.

The main script needs some sort of secrets storage so that it can be deployed somewhere other than my laptop and run on a cron.