profianinc / drawbridge Goto Github PK

We need to provide a CRUD for repository management, in-memory for MVP

In crates/name we're going to need to add following methods on <namespace> route:

• POST
• GET
• DELETE

Keys are Namespace, already existing in the crate and parsed from the URL and values are () for now.

Before routing the request to appropriate component, the name service should check that a repository identified by a given namespace exists and return a 404 if it does not.

We should also be able to match a user authenticated with x509, github, etc. in the routes.

We need to add a Github Action, which would verify that commits in the PR have been signed off

In #46 most of the functionality was removed

We should reenable the functionality and tests, as well as rely on and store Meta

Complete Meta is required on creation of tags and trees in the latest draft. We want to store Meta for all objects we store (tags, trees, repos).
It may greatly simplify the API and improve usability if this requirement would be (partially) dropped.
PUT of an object can (optionally compute) and return the complete Meta as a response.
Content-Type is always required, but Content-Length and Content-Digest can be computed after the body is received.

We could compute all digests and count the length (if those are not specified) and return that as a response. Client can then verify the digest and length to ensure correctness.

For tag service, for example, it means that users can create tag trivially even from command line via e.g. curl

@npmccallum what do you think?

Once we have a client library, we should write integration tests interacting with locally-served Drawbridge using the client library.

I think we should adapt app crate tests added in #90 once we have a client library and move them to top-level.

Tag names should be validated and enforce valid SemVer.

This should be implemented as a either tag service config, or a pluggable middleware.

Most request parsing errors are silently dropped and only a status code is returned to the API user.
Such behavior makes debugging difficult from user's perspective, therefore user-facing errors must provide a clear description of the error and it's cause along with the status code.
Partially addressed by #36 - once that PR is merged, this will mostly be a matter of going through Error::from_str invocations, which specify an empty string as the message argument and filling in the blanks

In #46 most of the functionality was removed.
Along that semantics have changed on how the service should work.

Currently, raw strings are returned

$ curl -v -X GET 127.0.0.1:8080/roman/dev/name/_tag/
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /roman/dev/name/_tag/ HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< transfer-encoding: chunked
< date: Fri, 22 Apr 2022 13:34:00 GMT
< 
* Connection #0 to host 127.0.0.1 left intact
v0.1.0v0.1.1

That should have been JSON

We should compute and set Content-Digest header for all responses containing objects

We should create a login page that can support several types of authentication schemes (github, google, etc) and redirect the user there if they are not authenticated.

The "name" term is overloaded

Closes #37

These headers refer to properties of the body of the request and NOT the desired properties of the response body.

For Content-Type, Accept is a reasonable alternative, but I need to do more research about the rest.

Currently most object contents are read completely in memory and returned as a byte vector. Contents should be streamed instead

Should support GitHub OAuth and use axum

Currently repository tree and tag stores are initialized on-demand with a write() lock, however, we should instead use a read() lock and first check if the store already exists, if not - relock with write() and initialize

Listing endpoints (currently only tags) should support pagination with the usual limit and offset URL parameters. We should also define a default limit, for example, 100.

On tree node upload, expected hash, stored in the parent, should be validated.

We would like to run Drawbridge inside of an enarx keep.

Repository service must check that a repository exists before passing the request further. See #63 (comment) for suggested approach

We should introduce storage traits shared between components

Assigning myself, since I already have a few drafts

Persistent storage will be needed for the various services. Likely files on disk. Some configuration and state items could be in a database (maybe SQLite?). File formats TBD.

Instead of using include_bytes!(..) to pull in an example key it would be preferable to pull it's path from the environment or generate it at runtime.

Once authenticated we can use the GitHub API to pull some basic information about the user. We should store this in their session for later use.

• Retrieve the users id from the GitHub API to be connected with a drawbridge user.

drawbridge-repo lib.rs is too big, we should extract Namespace, related parsing and testing logic to namespace.rs imported from lib.rs as use namespace::*

Now that we have drawbridge-http sub-crate containing functionality built on top of http crate, we should probably reexport http::* at the top-level of that crate (rather than providing it ::http prefix) to simplify usability of the crate.

Currently tag values are fully parsed and validated as a side-effect, we probably want to introduce a validation middleware instead for a cleaner approach

Provide an API for object storage, that client and server parts will be implemented against.

Current proposal is utilizing OpenAPI 3.1.0 https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.1.0.md which works with https://json-schema.org/

By utilizing OpenAPI we can benefit from tooling to generate documentation or even client and server implementations/boiler plate for different languages, e.g. see https://openapi.tools/

Some key points:

• Namespace: combination of username and optional groups
• Name: a name of a repository
• Tag: an immutable mapping of an arbitrary printable string, usually SemVer to a hash of a tree within the same namespace with the same name.
• Repository: a collection of trees and their tags
• Node: either a file or a directory, where directory is a collection of nodes.
  • Each node is write-once. After the node is uploaded once it cannot be reuploaded.
  • Each node has a Content-Type associated with it.
  • A node may belong to multiple trees
  • A directory node is a Merkle tree.
  • Each node has a hash associated with it, which is computed over it's contents including the metadata
• Tree: a node, usually a directory, but could potentially just be a single file.
  • Tree is uniquely identified by a combination of it's namespace, name and hash, which is computed over it's contents, it is therefore content-addressable
  • A tree hash can be also be uniquely identified by a combination of namespace, name and a tag
  • A tree can be yanked, which means that it cannot be downloaded anymore.

Tag service must check that a tag exists before passing the request further similarly to #67

Tag listing endpoint should support SemVer querying, e.g. GET myuser/myrepo?q=v1 would return all tags within v1 major.

We should introduce a custom MIME type for tree status and, when specified in Accept header, return on GET for trees.
The status message would contain at least:

• missing nodes

We should add some unit tests for routing

Support client authentication via a certificate signed by the server. This should be able to be applied to specific endpoints and should not prevent HTTPS connections to any unauthenticated ones.

Probably due to #79, uploads of root tree nodes are currently broken

$ curl -vL -X PUT -H "Content-Type: application/vnd.drawbridge.entry.v1+json" -d '{"digest":{"sha-384":"mqVuAfXRKap7bdgcCY5uykM6+R9GqQ8K/uxy9rx7HNQlGYl1kPzQho1wx4JwY8w="}}' 127.0.0.1:8080/roman/dev/name/_tag/v0.1.0/tree
*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> PUT /roman/dev/name/_tag/v0.1.0/tree HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.79.1
> Accept: */*
> Content-Type: application/vnd.drawbridge.entry.v1+json
> Content-Length: 89
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< location: /v0.1.0/tree/
< content-length: 0
< date: Fri, 22 Apr 2022 13:38:13 GMT
< 
* Connection #0 to host 127.0.0.1 left intact
* Issue another request to this URL: 'http://127.0.0.1:8080/v0.1.0/tree/'
* Found bundle for host 127.0.0.1: 0x138fa00 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host 127.0.0.1
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> PUT /v0.1.0/tree/ HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.79.1
> Accept: */*
> Content-Type: application/vnd.drawbridge.entry.v1+json
> Content-Length: 89
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< content-type: text/plain; charset=utf-8
< content-length: 17
< date: Fri, 22 Apr 2022 13:38:13 GMT
< 
* Connection #0 to host 127.0.0.1 left intact
Invalid namespace

$ curl -vL -X PUT -H "Content-Type: application/vnd.drawbridge.entry.v1+json" -d '{"digest":{"sha-384":"mqVuAfXRKap7bdgcCY5uykM6+R9GqQ8K/uxy9rx7HNQlGYl1kPzQho1wx4JwY8w="}}' 127.0.0.1:8080/roman/dev/name/_tag/v0.1.0/tree/
*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> PUT /roman/dev/name/_tag/v0.1.0/tree/ HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.79.1
> Accept: */*
> Content-Type: application/vnd.drawbridge.entry.v1+json
> Content-Length: 89
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< content-type: text/plain; charset=utf-8
< content-length: 16
< date: Fri, 22 Apr 2022 13:38:16 GMT
< 
* Connection #0 to host 127.0.0.1 left intact
Invalid tag name

We should provide an easy-to-consume and idiomatic, singular, Rust crate that can be consumed downstream.

Suggested structure:

• drawbridge - top-level module (crate drawbridge top-level)
  • {App, Client}::builder() - returns {App, Client}Builder
  • {App, Client}Builder::build() - returns {App, Client}
• drawbridge::{repository, tag, tree} - {repository, tag, tree}-related types, optionally possibility to directly instantiate an app or a client (at a later point)

Server-side and client-side functionalities are guarded by app and client feature flags respectively, both enabled by default. If both features are disabled - only types are available. At a later point we should also add a serde feature flag.

Usage flow:
App, implements tower::Service required to satisfy hyper's MakeService, that means that users of hyper can directly do a serve(drawbridge::App::builder().build()). That also means we're providing a generic tower::Service, so it can be used within any framework utilizing hyper, e.g. Axum.
AppBuilder provides config functionality to set up routes to serve services on, disable/enable services (by default everything is enabled on preconfigured routes) etc.

Similar for client

Probably due to our custom routing, redirects with trailing slashes or without are not properly handled in Axum:

$ curl -vL 127.0.0.1:8080/roman/dev/name/_tag
*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /roman/dev/name/_tag HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< location: /_tag/
< content-length: 0
< date: Fri, 22 Apr 2022 13:35:10 GMT
< 
* Connection #0 to host 127.0.0.1 left intact
* Issue another request to this URL: 'http://127.0.0.1:8080/_tag/'
* Found bundle for host 127.0.0.1: 0xab4a00 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host 127.0.0.1
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET /_tag/ HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< content-type: text/plain; charset=utf-8
< content-length: 33
< date: Fri, 22 Apr 2022 13:35:10 GMT
< 
* Connection #0 to host 127.0.0.1 left intact
Repository name must be specified

I think we should consider leaving "modularity" for a later point of time, and just have one Router with all the routes and go from there, which fix our issues here.