If (for whatever reason) the "sign request" endpoint fails, Evaporate silently fails. There is no indication in the UI that we were unable to sign the upload request URL.

May want to consider adding this at the same time we add upload progress indicators.

After initial upload screen we need to implement a complex workflow and save metadata to Django.

• Currently in Pane 1 we begin the upload immediately (assuming large files). We should evaluate if this is the desired functionality.
• Note that the progress bar does not show until the third Pane.
• Pane 2 implies that we can cancel uploads using the "x" button. If the file has already made it to S3, this will presumably delete this.
• Is the "Total Images" input actually useful or needed? It seems this can be determined from the number of uploaded files (though perhaps this is to deal with zip files).

To avoid name collisions in S3, we should not rely on the incoming file names. We should use a uuid for each file name, probably prepended with the user's id number.
uid-uuidv4
201-f47ac10b-58cc-4372-a567-0e02b2c3d479

Since we cannot assume that the file name is a secret we will need to be careful to enforce uniqueness.

By prepending the user id, we can ensure that even a malicious user will not be able to overwrite another user's data. The front end javascript code can generate the file name. Our signing URL should check to ensure that file name that the user asked to be signed for upload is properly prepended with that user's id. It should fail to sign the request if that is not the case.

This will require that the image model in Django is updated to have a metadata file name (the original file name from the user) as well as the S3 filename (uuid). We may want to allow the metadata filename to be changed.

I think this was a holdover for the bootstrapping of the app. Don't think there is any vector/draw requirements for this app.

https://github.com/azavea/raster-foundry/blob/develop/src/rf/package.json#L31

Users need to be able to register an account, and have a tracked dedicated amount of space for their account. When a user registers, they are given an API key that they need to be able to sign in and view. This API key will allow them to interact with the API server.

Outstanding questions:

• What should be the limit for a free account? My thought is 5G.

On the login page, if you click "Forgot?", "Send Activation Email", or "Sign Up" the background video resets. That transition used to be seamless.

Create a simple reusable React component for layers. This component is needed on the "My Imports", "Public Catalog", and "Favorites" tabs on the Library page.

Now that we've experimented with DRF ViewSets and sketched out the URL structure, actually implement and test the API.

See #5

The Layer API server will have the following endpoints. Interaction with any endpoint will require API key verification:

Create Layer

Given a list of images on S3 and additional metadata, tile those images into EFS and create a tile service endpoint. This endpoint returns success=true if the job is submitted successfully or success=false with an error if not.

Tooling will be built to allow for import into s3 from another non-public s3 bucket, the local file system or other storage as an intermediate step.

Example request:

{
    // REQUIRED: Name of the mosaic being created
    // MUST BE UNIQUE TO ACCOUNT.
    "name" : "nepal-imagery",

    // REQUIRED: List of s3 URIs to images.
    // Must be from a publicly accessible or raster-foundry owned bucket.
    // The order of the images is the priority of the images in the mosaic, with
    // images higher in the list being higher priority than images lower in the list.
    // So if a pixel is covered by more than one image, the image higher in the list
    // will have it's pixel shown.
    // Currently, these images have to be in RGB or RBGA format. If RGB, 
    // Pixels with value (0, 0, 0) are considered to be No Data.
    images : [
        "http://hotosm-oam.s3.amazonaws.com/356f564e3a0dc9d15553c17cf4583f21-9.tif",
        "http://hotosm-oam.s3.amazonaws.com/356f564e3a0dc9d15553c17cf4583f21-24.tif",
        "http://hotosm-oam.s3.amazonaws.com/356f564e3a0dc9d15553c17cf4583f21-20.tif"
    ]

    // OPTIONAL. Default: null. A text description of the layer. The
    // description can contain any legal character. Implementations SHOULD NOT
    // interpret the description as HTML.
    "description": "Nepal imagery from Digital Globe",
}

Example responses

{
    "success": true
}

{
    "success": false,
    "error": "Layer with the name 'nepal-imagery' already exists."
}

Layer metadata endpoint

Given a layer name, the status is either processing, failed with an error message message, JSON containing the metadata in TileJSON format with additional fields.

Example Request:

{
    "name": "nepal-imagery"
}

Example Responses:

{
    "name": "nepal-imagery",

    // One of: ["created", "failed", "processing"]
    "state": "processing"
}

{
    "name": "nepal-imagery",
    "state": "failed",
    "error" : "Unable to create mosaic due to account limits. In order to build this mosaic, please upgrade your account."
}

{
    "state": "created",

    // Date the layer was created
    "created" : "2015-07-25T00:32:22-04:00",

    // Location of simple leaflet web viewer
    "viewer" : "http://rasterfoundry.com/username/nepal-imagery/index.html",

    // The rest is taken from TileJSON spec: https://github.com/mapbox/tilejson-spec/tree/master/2.1.0
    // I have removed some optional elements of that spec that may be added later.

    // REQUIRED. A semver.org style version number. Describes the version of
    // the TileJSON spec that is implemented by this JSON object.
    "tilejson": "2.1.0",

    // The name of the layer (provided by the client in Layer Creation API call)
    "name": "nepal-imagery",

    // OPTIONAL. Default: null. A text description of the layer. The
    // description can contain any legal character. Implementations SHOULD NOT
    // interpret the description as HTML.
    "description": "Nepal imagery from Digital Globe",

    // REQUIRED. An array of tile endpoints. {z}, {x} and {y}, if present,
    // are replaced with the corresponding integers. If multiple endpoints are specified, clients
    // may use any combination of endpoints. All endpoints MUST return the same
    // content for the same URL. The array MUST contain at least one endpoint.
    "tiles": [
        "http://rasterfoundry.com/username/nepal-imagery/{z}/{x}/{y}.png"
    ],

    // An integer specifying the minimum zoom level.
    "minzoom": 0,

    // An integer specifying the maximum zoom level. MUST be >= minzoom.
    "maxzoom": 11,

    // The maximum extent of available map tiles. Bounds MUST define an area
    // covered by all zoom levels. The bounds are represented in WGS:84
    // latitude and longitude values, in the order left, bottom, right, top.
    // Values may be integers or floating point numbers.
    "bounds": [ -180, -85.05112877980659, 180, 85.0511287798066 ],

    // The first value is the longitude, the second is latitude (both in
    // WGS:84 values), the third value is the zoom level as an integer.
    // Longitude and latitude MUST be within the specified bounds.
    // The zoom level MUST be between minzoom and maxzoom.
    // Implementations can use this value to set the default location. If the
    // value is null, implementations may use their own algorithm for
    // determining a default location.
    "center": [ -76.275329586789, 39.153492567373, 8 ]
}

Delete Layer

This endpoint allows the user to delete a layer from the account.

Example request:

{
    "name": "nepal-imagery"
}

Example response:

{
    "success": true
}

Given that we expect file uploads for this project to be large, and we want to house raw input on Amazon S3, it may make sense to route user uploads directly to Amazon S3. In order to make this happen, we need to retrieve a signed URL from S3 via our backend, and thread it through to the JavaScript frontend.

A more detailed description, along with Python implementation details can be found here. That's more to understand the process though. Eventually, I think we'd want to lean on a JavaScript library like EvaporateJS that solves with problem for us, along with a custom Django route to provide the signed URL.

The main thing we trade-off here is the ability to do upload validation within Django. That may or not end up being acceptable.

The password change templates from registration redux need to be modified to match the style of the other RF registration components.

Wire-up prototype using JS

Also Favorites and processing (just static for now)

Add a Rest API for user registration and authentication. Try getting django-rest-auth to work, but if not, extract functionality from MMW and make it more generic so it can be reused more easily.

As outlined in #76 we will utilize Lambda to perform unzipping and image verification. After that is complete, a message should be posted to an SNS topic. That topic is responsible for posting a message to our application so that we can track all verified images before submitting the layer for geoprocessing.

We need an endpoint that can receive an SNS message containing a raster image ID and status. The structure of that request will need to be determined. Additional tables or other changes to the database structure may be needed to support this.

Figure out how Swagger integrates with DRF and start to formulate plans for REST API endpoints (See #6).

In the prototype there is a drag and drop area that should kick off file uploads to S3.

Wire-up prototype using JS

Though a password reset link is generated our front end does not display the proper forms and so the user never has an opportunity to reset the password.

Related to #63

Decide how to handle external file sources:

See right panel.

Wire-up prototype using Javascript

Incorporate new upload screen from updated prototype.
See #68

A python (debatable) client for the Raster Foundry APIs.

Requires configuration with an API key, set either through an environment variable, ~/.foundry config, or the like.

Commands

foundry ls

This command will list the names of a user's image layers.

foundry show [layer-name]

This command will describe the layer. If the layer has a processing status, show the user that fact, with a percentage complete if available through the API.

Otherwise, show the metadata described in #5.

foundry import [layer-name] [image uris]*

Import the set of images described in the URIs as a layer in the user's Raster Foundry account.
URIs can be local files or s3 URIs (to start).

foundry rm [layer-name]

Deletes a layer from Raster Foundry.

Prototype assumes, zip files can be uploaded to S3.

• Will raster processor work with zips directly?
• Should we unzip them and then upload them to S3?
• Can we unzip directly in S3?
• Should we drop this functionality?

Wire-up prototype using Javascript

Add ability to register a new user account and login.

Create a test spray service inside geop VM to test endpoint integration.

Currently, the favicon is the one used for MMW.

Using AWS Lambda and then store the unzipped files on S3

Actually call the login and logout endpoints from the client

Using GDAL

This should include a skeleton Django app for testing.

Create endpoint to associate file uploads with a registered user to restrict unregistered users from uploading.

Due to the fact that Layer Images are being directly uploaded from the client to S3, and it's an expensive operation to boot up a Spark cluster to begin processing, we should investigate ways to have a validation service determine if the files are worth further processing. The test may be as simple as detecting if GDAL can read it. Since we're allowing zipped files directly from the client, we could also piggy back the service to expand-validate-rewrite the zip files prior to processing.

Possible flow:
Upload to S3 -> triggers AWS Lambda job which tries to read (or unzips and reads) the Images -> sends or queues a message that the processing should begin (processing service) or that there are errors (app server).

Meet to discuss which things we should start/stop/continue doing based on past projects.

Based on the prototype and our plans for the API (See #5).

Modify scripts/test.sh so that it runs flake8 and unit tests. Once this is done, it might take some work to get the build to pass on staging.

This includes jobs to build the project and run unit tests for the develop branch and pull requests.

Create a toolbar for the layer selector component. Filter and search should be able to re-query layer data and redraw the component.

Using Swagger, swagger-codegen and docopt.

Wire-up prototype using Javascript

Create temporary geop VM and test spray service to test endpoint integration.