@wking
You seem have set up a reference engine by nginx and store some static page of ref-engine and cas-engine objects and debugged it. could you please submit them commit to contrib folder, or send to me? I'd debug on my local and implement golang client.

[From Aleksa]: Explain of Policy based well-known URL for discovery

发件人: Aleksa Sarai [[email protected]]
发送时间: 2017年8月30日星期三 19:33
收件人: xiekeyang
主题: Re: 答复: Discovery of OCI image

The pre-email content is rough, and I’m improving it. Now I’ve some
question about parcel want to discuss with you.

1. About discovery URI and distribution URI, are they same in your document?

No, they're not. They serve different purposes.

"Discovery URI" is something like opensuse.org/leap.
"Distribution URI" is more like
https://download.opensuse.org/some/project/some-manifest.json.

However, as I mentioned previously, I have a much nicer version of the document that uses "Template
Descriptors". It might be possible for the Distribution URI to be removed without losing the
functionality.

You choose below in doc,

https:///.well-known/cyphar.opencontainers.parcel.v0.json

but not https://coreos.com/.well-known/abd-index/com.coreos.etcd

what’s the advantages of cyphar.opencontainers.parcel.v0.json? You
drop URI path, is it OK and resolvable?

Parcel doesn't store any of the mappings or list of images in /.well-known/. There's only one JSON that
tells the client what the policy is for the domain for any arbitrary image. That can then redirect the
client to other JSON objects (the distribution objects) which then tell a user how to download the
particular image. If the client cannot find a distribution object by following the JSON policy, then it
breaks.

The reason why I didn't go with the same thing your proposal does is that your proposal makes the
assumption that the publishing/distribution system is able to modify /.well-known/oci-index/... on the
opensuse.org domain (for example).
Parcel doesn't need this requirement, all of the redirection and templating is done on the client
which means that there's no need to modify the /.well-known/ directory unless you're changing the
image policy.

I will try to finish my rework of the parcel proposal next week.

--
Aleksa Sarai (cyphar)
www.cyphar.com

If folks want to use it for occasional testing, I've published tremily.us/busybox#1.24.1 following these specifications (with the layer from here). Just don't hit me too frequently ;).

Here 1 declare that server provides discovered OCI index object which entry should conform to application/vnd.oci.image.manifest.v1+json. However in example casEngines entry 2 doesn't belong to
OCI index 3. We can't parse the object according to current image-spec 4.
How about to change the casEngine definition to annotations, to conform to OCI index strictly, like:

{
  "schemaVersion": 2,
  "manifests": [
    {   
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "size": 799,
      "digest": "sha256:e9770a03fbdccdd4632895151a93f9af58bbe2c91fdfaaf73160648d250e6ec3",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux"
      },  
      "annotations": {
        "org.opencontainers.image.ref.name": "1.0",
        "org.opencontainers.cas.protocol": "oci-cas-template-v1",
        "org.opencontainers.cas.uri": "https://example.com/oci-cas/{algorithm}/{encoded:2}/{encoded}"
      }
    }   
  ]
}

There is a problem: according to casEngines, it should be array type, but my example allows only one pair of protocol to uri. I think we can resolve this by extending annotations words to allow many pairs.
IMO, I feel only one pair of protocol and uri might be enough. Is there any possible of many uris for one protocol?

image-spec's image-layout is extremely similar to a special case of ref-engine discovery. I've stubbed out an image-spec branch to demonstrate this, which makes for a more compact and powerful layout spec (although it now depends on the additional ref-engine discovery wording from this repo). If this project, or something like it, ends up in an OCI Project, I recommend we move the layout spec out of image-spec and into this project (unless this project ends up being merged with image-spec).

One of the examples of increased power is the ability to shard the blob store (opencontainers/image-spec#449).

I was doing some initial work in this respect with casengine over the weekend (wking/casengine@252e064). What I have there is assigning a single file URI namespace based on a single filesystem directory or archive. That's obviously not going to scale well enough if a single well-known URI is referencing several archive files, and the consumer wants to access all of them. So, how to approach this? I see two possibilities:

a. Add an archive-file protocol to the ref- and CAS-engine registries (it can be a single protocol spec behind both entries) which says “here's an archive file. Once you retrieve it, treat it as a file URI namespace and use this child protocol inside”. For example:

{
  "protocol": "archive-file-uri-v1",
  "uri": "file:///var/lib/oci-image/app.tar.gz",
  "mediaType": "application/x-tar",
  "encoding": "gzip",
  "child": {
    "protocol": "oci-cas-template-v1",
    "uri": "file:///blobs/{algorithm}/{encoding}"
  }
}

The mediaType and encoding properties would be optional and unecessary for HTTP(S) URIs. Maybe optional for file URIs too, but providing them avoids the need for peek-inside client type detection. It's a bit unfortunate that we lean on tar here, because “tar” is a bit of a moving target (opencontainers/image-spec#342), and none of the options has a registered media type. Still application/x-tar is probably sufficient for these cases, because neither ref- nor CAS-engines need fancy file metadata, sparse files, or any of the other bits that are not supported in older tars.

b. Don't use file URIs. Insead, use fragments. A number of media types define these (text/plain, text/csv, application/pdf, application/xml, and possibly others), but application/zip does not (as far as I can tell), and tar has no registered media type at all. In this case, we'd have to define our own (unregistered?) tar type, and our own (unregistered?) fragment syntax for both tar and zip (we could probably use the same filename-based fragment syntax for both types). But we wouldn't have to define the archive-file-uri-v1 wrapping protocol. A CAS-engine entry would look like:

{
  "protocol": "oci-cas-template-v1",
  "mediaType": "application/x-tar",
  "uri": "https://example.com/image/app.tar.gz#blobs/{algorithm}/{encoding}"
}

And again, contentType and encoding may be useful options for protocols using file URIs.

I think I like (b) more, but I'd be fine with either approach. Do others have preferences? Other approaches to handling this?

Very early work in https://github.com/wking/casengine

Once this settles down, it probably merits a link from this repo. I've kept it separate because I want to see the CAS-engine registry landed in image-spec and image-tools (or a separate CAS-only project) under the OCI. It is very orthogonal to the rest ref-engines and ref-engine discovery stuff in this repository.

We have discussed and implemented OCI image discovery for some weeks. Currently we have a draft specification and experimental framework of discovery. While private discussion is becoming limited effect, we should consider to open a public discussion on OCI thread (I prefer @dev firstly), to talk if/when contribute this to OCI incubated project. Meanwhile we want OCI developers and users to express their opinions, making this project workable and acceptable by OCI ecosystem.

I present the initial demonstration as the below. If you think somewhere should be improved, or it is not the right time to discuss publicly, please put up directly.

Initial Discussion:

The main goal of project oci-discovery 1 is to demonstrate a mechanism to discover OCI images through a name delegation server by the below ways:

• Host-Based Image Names Delegation;
• Image Reference Engine Protocol;
• OCI canonical protocol template;
• Definition of merkle root object for a CAS discovery engine;
• An experiment of the discovery workflow;

As a supplement to OCI ecosystem, this just want to achieve canonicalization of image discovery and name delegation, and excludes topics of canonicalization about distribution out of the scope. We want that thread to be talked and addressed in another story.

While this project define OCI expected protocols, it allows consumers to register and make use of existing HUB based protocols such as Docker. Consumers just need to implement the protocol engine.

Contributing this to be an incubated project under OCI might be a considerable approach. It can help the iteration to going on, and make sure it on the right direction with developers contribution.

While there are some other good alternative such as parcel 4, ABD 5, etc, we may discuss here which is more slectable, or to take mixture among them. (During working on oci-discovery, we have get much great advice from the authors.)

@cyphar's distribution-uri ABNF used authority, but I used host in #2 because:

Where they're needed, I expect users to be providing userinfo and port information via other channels, especially since docker etc. currently use : to delimit tags.

In #5 (here) and #7 (here), @xiekeyang has wanted support for localhost:8080/…. I understand that is useful for testing, but don't think it should part of the host-based image name spec.

The .well-known URL lookup functions over both HTTP and HTTPS (e.g. see here). But if you are extracting the port from the image name, there's no way to know which protocol to use except by attempting both on that port. Trying to connect to a server using the wrong TLS protocol and finding out that you picked the wrong protocol (which is what would happened if we tried HTTPS and HTTP on localhost:8080) seems like a bigger issue than trying to connect to a server on the usual port for the protocol and finding out that the server is not listening on that port at all (which is what happens in our current protocol polling).

And I see no value to being able to specify explicit ports in production. Do we really expect production users to enter example.com:8080/… image names? Why would the example.com admins not be serving those images from 443 or 80?

There is a benefit for testing, because you don't need root on your host box to test if you can point the ref-engine discovery client at a high port. But I think we should find a way to work around that in testing (some ideas here, although we don't have anything trivial yet), instead of relaxing the production host-based image name spec.