If we're expecting users to serialize missing or null values into JSON as the javascript null object, Flask will parse it into None and pandas into a NaN.
If we need to be more permissive with missing data in JSON, we'll have to find/replace accepted NaN values.

Implement /forecasts endpoints in forecasts blueprint. Schema should also be updated with proper validation.

notes for validation:
will need to verify this, duration, and issue freq later
and issue_time. probably easiest to use pandas timedeltas for the former three, and a time function for issue_time

Originally posted by @alorenzo175 in #28

Currently, inserting values with the same timestamps throws a 1062 duplicate primary key error. We may need to add ON DUPLICATE KEY UPDATE to the INSERT statements in the store procedures for forecasts, observations and cdf_forecasts.

Add users to the database after signup. Perhaps this is a manual process, or there is a flag the admins have to select after the process described in #81

Looks like get obs/forecast data currently only supports JSON. It should also support csv.

https://dev-api.solarforecastarbiter.org/#tag/Observations/paths/~1observations~1{obs_id}~1values/get

https://dev-api.solarforecastarbiter.org/#tag/Forecasts/paths/~1forecasts~1{forecast_id}~1values/get

this coveralls.io.yml line needs to be changed or the test directory needs to be renamed to tests: https://github.com/SolarArbiter/solarforecastarbiter-api/blob/master/.codecov.yml#L27

I don't care which way we go. We're using tests in the core repo.

We should add more intro material that outlines how we expect most people to interact with the API: POSTing and GETing preexisting observations. An important item to clarify is one csv per observation (so one csv per data type) and therefore likely multiple POST/GET operations per site.

0.0.1a?

Don't want a disconnect to send a 500

Allow users to delete values at specific timestamps/ a list of timestamps

Forecasts and Oservations should not include the full nested site object, but rather a _links object with site attribute that links to site metadata.

including response schemas, request schemas, methods, routes

As documented here SolarArbiter/SolarArbiter.github.io#61

See https://stackoverflow.com/questions/47711689/error-while-using-pymysql-in-flask/48059894#48059894. Right now a connection is created for each request

as discussed here SolarArbiter/SolarArbiter.github.io#60 (comment)

Add a metadata link to csv header comments (as well as uuid) and _links attribute to json.

"Most users will interact with the API indirectly through actions on the dashboard."

dashboard links to dashboard.solarforecastarbiter.org, which doesn't exist. Can we instead link to @lboeman's new page proposed in SolarArbiter/SolarArbiter.github.io#52 ?

Since mysql fails silently when something doesn't exist, we should check for this and return something. Currently, deleting an object or trying to delete a missing object returns a 200 and trying to delete something you don't have access to throws a 404.

• create demo objects to act as static data store.
• Make storage module to load and interact with static data store.
• Update dashboard to call API for data.

issues discovered while working on dashboard:

• Observations lack interval_length field
• site_id stored as UUID when parsed by marshmallow. cast to string for storage.
• parse and store time duration variables in minutes
• don't allow duplicate timestamps in uploads.

including response schemas, request schemas, methods, routes

once the data model is redefined for observations

On top of existing Unit tests we need 'integration' tests to ensure the system functions as expected. the requests used to test the dev-storage backend #39 are a good place to start for these.

• storage creates a uuid for new metadata objects, adds 'Provider' field.

Should make sure users have validated email and that the organization has accepted the TOU before allowing access to login. Users should already require organization administrator approval since they will not have any default organization permissions.

Right now, the API docs say that a requests to /observations/id/values returns an array of dicts. The actual behavior returns just the dict as it should. We should update the docs and check them for the other values endpoints

including response schemas, request schemas, methods, routes

While working on #35 I found that APISpec now has a 1.0.0 release. Changelog can be found here: https://apispec.readthedocs.io/en/stable/changelog.html

Accept file uploads with the multipart/form-data content type for forecast and observation endpoints.

Need to address most of these before sending out stakeholder feedback request:

"Authorization request must include client_id='c16EJo48lbTCQEhqSztGGlmxxxmZ4zX' and audience='https://api.solarforecastarbiter.org'" can we make it look like this: "Authorization request must include client_id='c16EJo48lbTCQEhqSztGGlmxxxmZ4zX' and audience='https://api.solarforecastarbiter.org'" or similar?

The get site forecasts section says "Get all forecasts associated with site that user has access to". Probably better to say "Get metadata for all forecasts...". Same for get site observations.

get site information or metadata? probably metadata.

observation variables not up to date. Same for forecasts.

observation data example contains only 1 line. I guess it's ok but including a second line would make it more obvious that multiple data points are allowed in one upload.

The API has issue_frequency. The use cases describe this as run_length / issue frequency. run_length_issue_frequency?

Should we bring more/all of the forecast attribute descriptions into the API doc?

Interval length, not duration.

Intervals per submission is no more.

Add something like "to be determined in March/April 2019 to Reports and Trials".

We should validate things like site name better to avoid storing dangerous SQL injection or xss strings in the DB. Only allow letters digits and a few punctuation marks

Should check what kind of tracking type is set and validate that the required fields for that tracking type are present

As in SolarArbiter/SolarArbiter.github.io#62

When user has read permissions but cannot delete an object.

We'll want to define how errors are reported in responses so that responses are easily consumable (e.g. json returned to the dashboard for display). Probably worth implementing some custom exceptions and error handlers for Flask as shown here: http://flask.pocoo.org/docs/1.0/patterns/apierrors/

Let's list other possible characters that might be allowed in names. Some characters that I never want to allow are <>;@&=

For granting/revoking permissions. i.e. Read/write permissions on resources. This will have two aspects: Api endpoints and MYSQL triggers.

All endpoints should respond with:

• a 200 code when successful
• 401: unauthorized when a user is not logged in (this may need to be merged with 404 depending on if we require signup for public data.
• 400 when a logged in user has permissions to perform an action but the request is invalid for some other reason. This usually produces a meaningful error message.
• 404 User lacks permissions or the resource does not exist. This is so that users cannot discover resources they do not have access to.

Many fields in the schema are not required but have no missingor default values. We should probably set these fields so we know the once place where default values come in. The database storage_interface also won't work as is since the fields are not set at times.

Do they each have a blueprint? Does one post the uuids of things that need to be compared?