There's quite a bit of design work in this one before we decide on an implementation:

ADAMA should be able to send back developer-specified provenance/attribution with its responses. In our service types that return JSON, this may need to be an extra entry in the response document. In services where we don't control the response content, we may consider injecting a custom Header into our response.

We need to think about the ux of cli tools, and whether/how to integrate with Agave cli.

As part of our attempt to create automatically documenting APIs and do input validation, I propose to extend the metadata scheme for ADAMA to allow publishers to specify parameters and metadata, much like we do for Agave /apps parameters. Ideally, we'd use near-identical metadata schemes to promote compatibility with Agave core APIs.

A generic adapter is a generalization of a query or structured adapter, in which the adapter can return content type other than JSON (for example: images or CSV files).

Brought by Alex in all-hands meeting:

A map-concat adapter generalizes a map-filter. The drawback is that the function has to return a list of length 1 for the most usual case of just map.

Alternatively, we could support it as another type, i.e., having map-filter, and map-concat as different types, and letting the user choose.

We need to allow the database to be backed up and restored from scratch. This also means reconstructing the images for the containers and relaunching workers.

Think about using the same model as Agave, or directly using the /notifications service of Agave.

The following request raises an exception

wmoreira@wmoreira-mac:~/Sandbox/swagger-ui/dist (master●) (ADAMA) $ curl -skL -H "Authorization: Bearer $TOKEN" http://api.araport.org/community/v0.3/aip/atted_coexpressed_by_locus_v0.1/search\?locus\=AT5G49450\&threshold\=0.1\&relationship_type\=correlation_coefficient
{"result": [
{"relationships": [{"direction": "undirected", "type": "coexpression", "scores": [{"correlation_coefficient": 0.8824}]}], "related_entity": "AT5G49448", "class": "locus_relationship", "locus": "AT5G49450"}
, {"relationships": [{"direction": "undirected", "type": "coexpression", "scores": [{"correlation_coefficient": 0.7257}]}], "related_entity": "AT1G02610", "class": "locus_relationship", "locus": "AT5G49450"}
, {"relationships": [{"direction": "undirected", "type": "coexpression", "scores": [{"correlation_coefficient": 0.5754}]}], "related_entity": "AT2G05540", "class": "locus_relationship", "locus": "AT5G49450"}
, {"relationships": [{"direction": "undirected", "type": "coexpression", "scores": [{"correlation_coefficient": 0.6867}]}], "related_entity": "AT5G56550", "class": "locus_relationship", "locus": "AT5G49450"}
, {"relationships": [{"direction": "undirected", "type": "coexpression", "scores": [{"correlation_coefficient": 0.6903}]}], "related_entity": "AT3G15450", "class": "locus_relationship", "locus": "AT5G49450"}
, {"relationships": [{"direction": "undirected", "type": "coexpression", "scores": [{"correlation_coefficient": 0.6109}]}], "related_entity": "AT1G49500", "class": "locus_relationship", "locus": "AT5G49450"}
, {"relationships": [{"direction": "undirected", "type": "coexpression", "scores": [{"correlation_coefficient": 0.665}]}], "related_entity": "AT1G70290", "class": "locus_relationship", "locus": "AT5G49450"}
, {"relationships": [{"direction": "undirected", "type": "coexpression", "scores": [{"correlation_coefficient": 0.568}]}], "related_entity": "AT5G54540", "class": "locus_relationship", "locus": "AT5G49450"}
, {"relationships": [{"direction": "undirected", "type": "coexpression", "scores": [{"correlation_coefficient": 0.665}]}], "related_entity": "AT2G18700", "class": "locus_relationship", "locus": "AT5G49450"}
, {"relationships": [{"direction": "undirected", "type": "coexpression", "scores": [{"correlation_coefficient": 0.6209}]}], "related_entity": "AT3G62950", "class": "locus_relationship", "locus": "AT5G49450"}
, {"relationships": [{"direction": "undirected", "type": "coexpression", "scores": [{"correlation_coefficient": 0.6621}]}], "related_entity": "AT5G22920", "class": "locus_relationship", "locus": "AT5G49450"}
, {"relationships": [{"direction": "undirected", "type": "coexpression", "scores": [{"correlation_coefficient": 0.5292}]}], "related_entity": "AT1G80630", "class": "locus_relationship", "locus": "AT5G49450"}
, {"relationships": [{"direction": "undirected", "type": "coexpression", "scores": [{"correlation_coefficient": 0.506}]}], "related_entity": "AT3G15358", "class": "locus_relationship", "locus": "AT5G49450"}
, {"traceback": "Traceback (most recent call last):\n  File \"/root/worker.py\", line 27, in callback\n    t = self.operation(message)\n  File \"/root/worker.py\", line 44, in operation\n    getattr(self.module, endpoint)(json.loads(body))\n  File \"/root/user_code/main.py\", line 59, in search\n    agi_locus_from_entrez = resolve_locus(result['gene'])\n  File \"/root/user_code/main.py\", line 80, in resolve_locus\n    result = response.json()['results'][0]['data'][0]['row'][0]\nIndexError: list index out of range\n", "error": "list index out of range"}
],
"metadata": {"time_in_main": null},
"status": "success"}

We should not allow an empty name:

$ curl -k https://adama-dev.tacc.utexas.edu/community/v0.3/namespaces -H "Authorization: Bearer ...."  
{
    "result": [
        {
            "description": "Developer workspace for @", 
            "name": "", 
            "url": null
        }, 
        {
            "description": null, 
            "name": "walter-dev", 
            "url": null
        }, 
        {
            "description": "Developer workspace for @mwvaughn", 
            "name": "vaughn-dev", 
            "url": null
        }
    ], 
    "status": "success"
}

A pass-through adapter just forward requests and responses.

We want to make it easier for developers to specify errors in their adapters. Adama could provide a fail function that aborts and returns user provided information through the standard error channel. The function can be injected into the namespace of the adapter, so the user doesn't have to import any extra module.

map_filter is the proper name to the pattern. In addition, it's easier to understand that we can do maps, filter, or both at the same time.

We want to fail soon at registration time if the user's code has syntax errors or missing requirements. Currently, this happens at the time of the first request.

After creating a namespace, a user should be able to update any field save for 'name' by issuing a PUT to the namespaces endpoint

Following the app generator for front end apps, it would be nice to have an adapter generator that builds a complete template for the several types of adapter. Using the git checkout feature, the template would include all the metadata necessary for registering the adapter.

In the future, this generator could be integrated with the app generator, so developers can have everything in one package.

The status endpoint should check whether the workers for a service are really running. There should be an option to restart them, or to manage them. Not clear who would have access to such features: the user who register the adapters? Or only Adama admins?

Issue #23 also needs the restart (and rebuild, in case images are lost).

From Matt V.:

I wanted to chat about the case where a service is moved (or copied) from one namespace to another. Potential use cases for this involve turning over “maintenance” of a service to someone else, moving services out of a “development” namespace into a “production” namespace (vaughn vs vaughn-dev for example), or moving services to a namespace owned by the AIP (or other tenant, someday) project as part of a “publication” process.

The REST implementation seems simple: Via a PUT operation on a known service, allow update of the “namespace” attribute as long as the user has write access to that namespace. The question becomes what happens to the previous instance of the service. I would argue that we need to keep it around, but resolve it with a 301 Moved Permanently unless someone explicit issues a DELETE on the old resource.

/community/vaughn-dev/starbuck

moves, after PUT operation, to

/community/vaughn/starbuck

GET to /community/vaughn-dev/starbuck returns

301 Moved Permanently
Location: /community/vaughn/starbuck

until.. I DELETE /community/vaughn-dev/starbuck

Which then returns

404 Not Found

Adama should support CORS headers.

A passthrough adapter operates via an /access endpoint, similar to how the other adapters operate via /search and /list. This leaves the / endpoint free for returning a representation of the Adama service object.

ADAMA should be publishing a Swagger spec for its core abilities (namespaces, services) but also, we should see if there's a simple way for devs to specify either a) Swagger or b) sufficient metadata that we can roll usage documentation from community APIs into ADAMA's documentation. In combination with the master Agave swagger spec, this should allow the App builder SDK to display and interpret usage info about community APIs. This should be VERY powerful.

ADAMA should check and reject invalid URIs at POST/PUT time rather than accepting the request and failing internally. For example, submitting a bad URL as a git_repository field results in an attempt to check out that repo and generates a nasty stacktrace error. Another example, when one provides 'url' as a metadata field, it is not checked for validity. This could be a security risk.

When registering a passthrough adapter using git, Adama complains of a missing main module, which shouldn't be necessary.

The json returned on error contains the full traceback, making the output long and scary. Should we allow an option verbose true/false to enable/disable this behavior? If so, what would be the default value?

Adama should provide a way to specify branches, tags, or any tree-ish object, for checking out when registering or updating a service via git.

If a map adapter raises an exception, we want to immediately abort the request and return the traceback. Currently, the adapter keeps being applied to every JSON object, and an array of errors is returned:

{"result": [
{"error": "integer division or modulo by zero"}
, {"error": "integer division or modulo by zero"}
],
"metadata": {},
"status": "success"}

Multiple adapters may try to update iptables at the same time, and iptables holds a lock. Probably not a big deal since registration is a low frequency event, but may affect automatic tests.

In addition to the code, we also want to support a file metadata.json with the specification of the parameters for registration. The main idea is to allow the registration with the url of the git repository as the only piece of data.

For cases when a map adapter does an unauthenticated request to a third party service, we don't want to expose the bearer token used to access Adama.

This may be a bug or a future. Since the namespace is part of the full name of the images for the containers, they cannot contain uppercase. If we don't like this, we may need to add a mapping between user names and internal names. This is related to issue #3.

The endpoints listed in the response should depend on the type of adapter (for example, /access for passthrough adapters).

A map adapter currently applies a transformation to an array of objects and returns the result as an array with label "results" (usual Agave 3-stanza format).

It is a valid use case to preserve the structure of the original response of the third party service. This would allow to use the Adama url of the registered service in place of the original service.

Pros:

• Repositories of adapters easier to understand
• self documenting
• Allows more than one adapter in same directory

Cons:

• One more option (but this is mitigated when #4 is implemented)

People seem to use .yaml and .yml when working with YAML files, so we want to support both.

Swagger docs should be served by an outside of WSO2 url. We'll use adama-dev.tacc for now, until deciding a proper place.

Follow HATEOAS and return already built urls to access the namespace and service.

Need to define where to upload backups for storage, and frequency of backups.

Eventually, query (and perhaps map_*) adapter types should support (and maybe even enforce) validation of JSON responses against a) AIP-defined JSON schemas b) External JSON schemas. This is pretty easy to code up in the adapters themselves but I'd like to promote this as core ADAMA functionality. We're a ways off from needing this since we haven't even defined our schemas yet.

See https://pypi.python.org/pypi/jsonschema for more information on a jumping off point towards implementation.

In particular, this means Adama should check the JWT in the requests.

From Matt V.:

After you implement the additional mediator types, we will need a security model in place for ADAMA. I think we discussed this as our initial draft

1. All namespaces are publicly discoverable after creation (consistent with Adama’s behavior now)
2. A namespace can be created by any user. It may only be edited or deleted by that user
3. All services are are publicly discoverable after creation (consistent with Adama’s behavior now)
4. A service can be created by any user. It may only be edited or deleted by that user

In the longer run, we will want to adopt Agave’s ACL model where

READ: Discover metadata about the service or namespace
WRITE: Update or delete the service or namespace
EXEC: Query against the service. Has no meaning for namespaces.

We want to maintain the case in the names of services and adapters for displaying. But we should support case insensitivity in urls and in inputs from users.

The worker containers die when their persistent connection to the queue is interrupted. Make them so they can reasonably retry a number of times. This would make them resilient to temporary queue failures.

This service automatically detected the whitelist, and in doing so duplicated the apps.araport.org entry. I assume this is a more pervasive issue. The functionality of the service was not affected by this bug.

curl -sk -L -X GET $API/$NS/services -H "Authorization: Bearer $TOKEN"
{
    "result": [
        {
            "code_dir": "/tmp/tmpYeTTFX/user_code", 
            "description": "Query Thalemine", 
            "json_path": "", 
            "language": "python", 
            "main_module": "main.py", 
            "metadata": "", 
            "name": "query_thalemine_as_jbrowse_index", 
            "namespace": "$NS", 
            "notify": "", 
            "requirements": [], 
            "type": "map_filter", 
            "url": "https://apps.araport.org/thalemine/service/jbrowse/names/3702", 
            "version": 0.1, 
            "whitelist": [
                "apps.araport.org", 
                "apps.araport.org"
            ], 
            "workers": [
                "b78265f520a66265685f49ae52e7bc5c53551e3f913454766614c609a2983445"
            ]
        }
    ], 
    "status": "success"
}

Configure to create a handful of workers for each service, to support faster queries and fault tolerance. It will also give us an idea about how the VM behaves under heavier load from containers.

Example output:

curl -H "Authorization: Bearer $TOKEN" -sk -X POST https://api.araport.org/community/v0.3/namespaces 
-Fname="vaughn" -Fdescription="API publication space for mwvaughn"

{
    "result": "http://dev.adama.tacc.utexas.edu:80/vaughn", 
    "status": "success"
}

My expectation is that the result would return a URL referencing api.araport.org

We want to keep documentation about the source repo, to allow the re-creation of the service or for attribution purposes. The state endpoint should return such information.

If the check-out of a repository takes too long, WSO2 times-out. We need to do the check-out asynchronously to avoid this. We'll return a temporary url with a hash or token for the user to check the progress, since we won't know the metadata of the adapter until the repo is cloned.

Instead of doing a full DELETE and re-POST to update a service from its Git repo, one should be able to issue a POST to invoke refresh behavior or modification of service metadata

From Jason Miller "Third party Web Services would like to track their usage. If AIP Science Apps use their services, all their users will appear to be at AIP. We should pass through other information such as the source of the Science App or IP address of the user. This idea came up in our discussion with Nick Provart, whose web services do track usage."