privatekit / safepaths-datascience Goto Github PK

At the moment utils.py only recognizes .csv and .json. While I think those will account for most dataset, we might want to implement the Google API calls for google spreadsheets as well.

The "check" functions in the utils.py should log their status (both positive and errors)

At the moment the data ingestion script do not ingest timestamps or similar. The problem is that public datasets will have different formats.

1. We need something that recognizes the date/timestamp format (without loading heavy packages as pandas, if I understand well) in each dataset we ingest
2. What format we implement eventually? POSIX, I imagine.
3. This will be fundamental to periodically clear the data every 15 days

How do we make sure that public data do not overlap and numbers are summed up?

To be taken into consideration: the public data I am currently pulling do not have mobile GPS data, but refer to reference pinpoint coordinates (i.e. one central point for each state/province, or the coordinates of the hospitals).