If you see a shelter listed in https://github.com/simonw/irma-scraped-data/blob/master/florida-shelters-missing.json which is NOT actually missing from https://irma-api.herokuapp.com/shelters paste the details in here as a comment and the bot will know not to flag it as a potentially missing shelter.

links are not working in line 3 and 9

This repo now contains the (revision-tracked) output of a number of different scrapers.

We have a team of people working to keep https://irma-api.herokuapp.com/ up-to-date - that's the dataset that powers https://www.irmashelters.org/

We have avoided automatically adding shelters to irma-api because the data sources are often unreliable, and we feel it is better to have a human being manually check each entry, avoid duplicates, clean up the data and maybe use other sources to confirm that each shelter exists and is open.

BUT... there's no reason we can't have automated tools make suggestions to our human editors.

Here's the task: write code which pulls the current list of shelters from https://irma-api.herokuapp.com/api/v1/shelters and then pulls one of the scraped JSON files from this repo - then compares the two and tries to detect shelters that are missing from irma-api.

A useful starting point would be this file, which is scraped from http://www.floridadisaster.org/shelters/summary.aspx - the most recent JSON can be loaded from here: https://raw.githubusercontent.com/simonw/irma-scraped-data/master/florida-shelters.json

A useful script would do the following:

• Retrieve the current list of shelters from https://irma-api.herokuapp.com/api/v1/shelters (or from the mirror of that file in GitHub: https://raw.githubusercontent.com/simonw/irma-scraped-data/master/irma-shelters.json )
• Pull the latest scraped data from https://raw.githubusercontent.com/simonw/irma-scraped-data/master/florida-shelters.json
• Loop through the list in florida-shelters.json and, for each shelter, try to determine if it is already listed in irma-api. My hunch is that the best way to do this is by comparing latitude/longitude points, since the names and addresses of the shelters frequently have different spellings. A fuzzy text match against the name or address might work well too - try different things and see what works out. I've been using a 7 character Geohash of the latitude/longitude for dupe detection elsewhere and it seems to work very well.
• Output a list of possible dupes. Share that with the irma-response-slack.herokuapp.com community in the #shelters channel - manually at first, but once it's proved itself set it up to automatically share in some way.

The way my irma-dupe-detection script works may be a useful inspiration: https://github.com/simonw/irma-scrapers/blob/59cf9906d1005972ab2e172ec494336b7b8b8434/irma_shelters.py#L174-L217