trustbloc / edv Goto Github PK

Make a reusable client type that can be used to interact with an EDV in code. The client will be responsible for managing the REST API calls, encryption keys, etc.

Part of #22

Use the new common logging library that was added to edge-core.

Just need to add "EDV_" in front of all the environment variables names in start.go.

In the updated spec (26 January 2020), they changed the structured document ID to be a mandatory base58-encoded 128-bit random value. Currently the code doesn't enforce any sort of value for the ID, but it needs to per the spec.

Implement Encrypted Index support.

To be in line with the style used in the official Go project.

The Stream section would be lower priority.

https://digitalbazaar.github.io/encrypted-data-vaults/#data-vault-https-api

This issue was generated by todo based on a TODO comment in eeb06b1 when #1 was merged. cc @DRK3.

Once trustbloc/edge-core#26 is done, use it in EDV to avoid unnecessarily recreating indices.

Implement the "E" in "EDV": Store EncryptedDocuments instead of StructuredDocuments.

Add the ability to publish a Docker image.

It's a bit weird that the store is passed into the EDVProvider.CreateDocument method, since the EDVProvider has access to all of the stores already.

Right now it just uses a precalculated value. Ideally it should be updated to show how you can create an indexed attribute from scratch using HMAC-SHA256.

Right now the EDV only outputs a message to let you know it's running, but it would be helpful to have messages when other veents happen (such as a new document being created).

New BDD tests will also be required to demonstrate this functionality.

add swagger annotations for OpenAPI.
add Makefile target to generate OpenAPI/swagger documentation.

Currently you must provide a value in addition to an index name. Also add support for searching for any document that contains the indexed attribute name as well, as described in this example: https://identity.foundation/confidential-storage/#example-35-data-vault-query-for-a-particular-attribute-name

The EDV spec does not mandate a particular authorization scheme. In our implementation here, it's essentially left up to the storage provider. e.g. for CouchDB, the username and password can be supplied via the database URL flag.

We could use DIDs and signing, similar to how Transmute does in https://www.youtube.com/watch?v=1zNe_KsdszI to authenticate users.

Currently the only check is a basic JSON unmarshal call. This means that the only check the server will do right now is validating whether the input is valid JSON, but not whether it's actually got the correct data per the spec. Implement robust validation.

Additional check: return an error if an "invalid id" is provided. The ID has to work correctly when concatenated to a URL. Any provided ID that will cause this to fail (such as ones that contain //) need to generate an error. No longer applicable since vault IDs are generated by the server, and document IDs have to be Base58-encoded.

Some responses (like error messages and log spec change response) aren't in JSON format. It would be better if all response body messages were in JSON format for consistency.

Also ensure that the Swagger UI is updated to expect JSON as well.

• Add documentation for 0.1.1
• Release 0.1.1

Once hyperledger-archives/aries-framework-go#1409 and the NIST-approved crypto stuff is done, update the code in the BDD test accordingly.

Mock EDV APIs for storage to allow plugging Implementation later on.

• Basic API implementation, simple in-memory store trustbloc/edge-store#27
• Implement EDV client #3
• EDV Docker image #14
• Client BDD test

(No encryption or authentication, using StructuredDocuments instead of EncryptedDocuments for now)

The authorization mechanism is left up to the implementer.

From the spec:

It is necessary to have a mechanism that enables authorized sharing of encrypted information among one or more entities.

The system is expected to specify one mandatory authorization scheme, but also allow other alternate authorization schemes. Examples of authorization schemes include OAuth2, Web Access Control, and Authorization Capabilities (ZCAP-LD).

Currently, in our implementation it's essentially left up to the storage provider. e.g. for CouchDB, the username and password can be supplied via the database URL flag.

We could use DIDs and signing, similar to how Transmute does in https://www.youtube.com/watch?v=1zNe_KsdszI to authenticate users.

There are additional properties in DataVaultConfiguration objects that haven't been implemented, such as sequence, controller, delgator, etc.

However, there aren't a lot of details on these fields and there still seems to be some things in the spec that need to get worked out since there's a note in that section that says:

ISSUE 5 Data vault configuration isn't strictly necessary for using the other features of data vaults. This should have its own conformance section/class or potentially event be non-normative.

We should figure out what to do here, perhaps after the spec has been updated?

See https://www.youtube.com/watch?v=1zNe_KsdszI to see how Transmute uses a DID to verify that the controller is indeed the one accessing the vault.

Implement the Update Document and Delete Document endpoints.

Also make sure it's thread safe in case there are multiple clients accessing the EDV at once.

The EDV specification allows for an encrypted index+value pair to be declared as unique. Add support for this.

These all happen sequentially in code. If either of these requests fails, then the database will be left in a weird state.

trustbloc/edge-core#4 must be done first

Then we can

1. Update Edge-Core in go.mod to the new version
2. Add a parameter to the command line to allow EDV to use the in-memory store or a CouchDB store.

Similar to https://github.com/trustbloc/edge-service/pull/354/files#diff-97e884010a415abaca46e2c19efb838bR617

Implement https://digitalbazaar.github.io/encrypted-data-vaults/#discovering-service-endpoints.

• Add documentation for 0.1.2
• Release 0.1.2

If your vault ID or document ID contains a slash, the resulting location that will be returned from the server will be unreachable since the slash breaks the path. Even if the slash is urlencoded as %2F, it still can't be reached.

This seems to be an inherent issue in the Go net/url and so this issue carries over to the Gorilla mux package. See gorilla/mux#77 for details.

There is a workaround: call mux.useEncodedPath() before starting the router.

We will also probably want to urlencode the returned location in the response so that it's actually usable as well.

The error message that comes up is odd: Failed to create a new data vault: store not found.

See if it can be fixed. If it's a limitation of CouchDB/kivik (which I suspect it is), then instead provide a better error message to the user.

May or may not be an issue with go-swagger - I was unable to get it to actually show the location header, despite it showing up in the expected response. This issue affects the create vault and create document endpoints in the OpenAPI demo.

When edge-sandbox was running on AWS, there was an intermittent failure that happened when edge-service was retrieving a credential from the EDV. The EDV queried the underlying CouchDB database (which was running in cluster mode) and it returned no documents. This seemed to happen more often where there were many CouchDB documents in the store (over 50). When inspecting the database, the documents appeared to be there, so it's strange that CouchDB returned no results.

Things to investigate...

Single node vs cluster mode - does that make a difference?
When those failures happen - check and confirm that the query is correct.
Could there be an issue with the underlying Kivik library?
Checks the CouchDB logs.