oras-project / oras-go Goto Github PK
View Code? Open in Web Editor NEWORAS Go library
Home Page: https://oras.land
License: Apache License 2.0
ORAS Go library
Home Page: https://oras.land
License: Apache License 2.0
Since we are decoupling containerd
, we need a new interface for Target
.
type Target interface {
remotes.Resolver
}
oras-go v0.5.0 has been validated in the Helm codebase (see helm/helm#10294)
IMO this was a primary blocker for releasing this library with a v1.0 version.
Now that we know it works, more comfortable with doing so.
When running multiple helm chart
commands at the same time, there can be a race to write the updated OCIStore index reference resulting in missing references for later commands. Is it expected that this should work or should each helm chart
command be given a new registry cache directory to avoid this?
Failing test case:
package content
import (
"io/ioutil"
"testing"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
)
func TestConcurrentSaveIndex(t *testing.T) {
tempdir, err := ioutil.TempDir("", "")
if err != nil {
t.Fatal(err)
}
store, err := NewOCIStore(tempdir)
if err != nil {
t.Fatal(err)
}
err = store.LoadIndex()
if err != nil {
t.Fatal(err)
}
parallelStore, err := NewOCIStore(tempdir)
if err != nil {
t.Fatal(err)
}
err = parallelStore.LoadIndex()
if err != nil {
t.Fatal(err)
}
store.AddReference("my-first-reference", ocispec.Descriptor{})
parallelStore.AddReference("my-second-reference", ocispec.Descriptor{})
err = store.SaveIndex()
if err != nil {
t.Fatal(err)
}
err = parallelStore.SaveIndex()
if err != nil {
t.Fatal(err)
}
err = store.LoadIndex()
if err != nil {
t.Fatal(err)
}
parallelStore.LoadIndex()
if err != nil {
t.Fatal(err)
}
if _, ok := store.ListReferences()["my-first-reference"]; !ok {
t.Error("Store does have have my-first-reference")
}
if _, ok := store.ListReferences()["my-second-reference"]; !ok {
t.Error("Store does have have my-second-reference")
}
if _, ok := parallelStore.ListReferences()["my-first-reference"]; !ok {
t.Error("Store does have have my-first-reference")
}
if _, ok := parallelStore.ListReferences()["my-second-reference"]; !ok {
t.Error("Store does have have my-second-reference")
}
}
Besides #52, we also need to implemented the Extended Copy API.
Fetch
to get the artifact manifestAs a developer, I want oras go to have Retag
function so I rename a tag. Function signature would be like
ReTag(ctx context.Context, oldReference string, newReference string) error
To make Copy
useful, we need to implement a Memory
target based on the current oras.land/oras-go/pkg/content.Memory
.
It is good to have examples in the code. However, the examples introduce extra dependencies in the go.mod
and go.sum
files, which are not used by the actual oras-go
library.
It will be better if we can separate the dependencies required by the library and the examples. When it is done, we have even more complex examples without bothering the go.mod
file for the oras-go
library.
To remove the unnecessary dependencies not related to the oras-go
library, all we need is to add a go.mod
file under the examples
folder.
Reference: Multi-Module Repositories.
The readme notes:
๐ง This project is currently under active development. The API may and will change incompatibly from one commit to another. ๐ง
This is a blocker for using this package as a trusted source and releasing stable functionality based on it. Is there a plan to a 1.0.0? Will there be any API guarantees at any point coming up?
People want to put artifacts in OCI registries but, I think, that needs to happen based on a stable foundation. What's the plan to get there?
I have found that FileStore
implements a buggy Close
function:
Lines 306 to 316 in 99e2fde
https://go.dev/play/p/UvfCFZTUwpk
package main
import (
"fmt"
"log"
"strings"
"github.com/pkg/errors"
)
func main() {
errs := []string{}
err := errors.New(strings.Join(errs, "; "))
if err != nil {
fmt.Printf("This is returning an error due to %v\n", errs)
log.Fatal(err)
}
}
Run
This is returning an error due to []
2009/11/10 23:00:00
Only users handling errors in the deferred function:
# not affected
defer store.Close()
# affected
defer func() {
err := store.Close()
# e is the named returned variable of the function
if e == nil {
e == err
}
}()
defer func() {
err := store.Close()
# e is the named returned variable of the function
if e == nil && err != nil && err.Error() != "" {
e = err
}
}()
Check errors slice before returning a new (valid) error:
// Close frees up resources used by the file store
func (s *File) Close() error {
var errs []string
s.tmpFiles.Range(func(name, _ interface{}) bool {
if err := os.Remove(name.(string)); err != nil {
errs = append(errs, err.Error())
}
return true
})
if len(errs) == 0 {
return nil
}
return errors.New(strings.Join(errs, "; "))
}
Based on #50, the Copy
API originated by #8 need to be re-implemented without containerd
.
That is, we need our version of
images.Childen
images.Dispatch
Since we are implementing the images.Dispatch
, we can optimize the graph traversal for our ORAS scenarios.
In fact, images.Dispatch
is designed for trees not for DAGs. It is less inefficient if there are multiple nodes pointing to the same node. The pointed node might be transferred multiple times at the same time based on the concurrency level.
In the current code, we also implemented something equivalent to remotes.PushContent
. Although the layers are transferred in parallel, the manifests are transferred in sequences. It might not be efficient if we want to copy a manifest list or index.
I don't know if I'm doing anything wrong, but I've used oras.WithManifestAnnotations()
to push my artefact with some metadata, that worked well because if I curl the repo directly I see it there:
$ curl -sL -u"user:pass" https://my-repo/v2/oso-plugins/manifests/oso-legacy | jq .
{
"schemaVersion": 2,
"config": {
"mediaType": "application/vnd.unknown.config.v1+json",
"digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
"size": 2
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar",
"digest": "sha256:8e985dc2c81bd5e8bfafb2858b9a9cb420c02a29d5a96f32eb00f5850556410a",
"size": 755,
"annotations": {
"hook": "chmod +x ~/.oso/bin/oso-legacy",
"org.opencontainers.image.title": "oso-legacy"
}
}
],
"annotations": {
"caveats": "to get auto-completions do this...",
"description": "a bridge to the past",
"shortDescription": "a plugin"
}
}
But then when I pull it using desc, artefacts, err := oras.Pull(...)
I can loop through the artefacts and get the annotations from each of them, but desc.Annotations
is empty...
The result of fmt.Println(desc.Annotations)
is:
Annotations: map[]
As a part of the discoverers works, the oras project is defining the work to support graph copy from on remote registry to another.
The copy API semantics is about targets and if local is a target this doesn't work.
For e.g something like this
oras copy --recursive myregistry/net-monitor:v1 <local_target>
Hi! Is there planned support in v1 for local OCI content stores?
We've been using it in https://github.com/opcr-io/policy - version 0.4.0.
Create nested module oras.land/oras-go/pkg
for the old implementation and leave oras.land/oras-go
to the new implementation. Hence, we can have the old and new implementations at the same time.
The oras-go
library is supposed to be simple and reliable. At the same time, we also want to do experiment on new APIs / features. Since oras-go
is library, not a CLI, we can not use environment variables like ORAS_EXPERIMENTAL=1
to turn on the experimental APIs. It becomes an issue on how stable and experimental APIs / features are managed.
We can have 2 secured branches main
, and experimental
, and develop individually. On release, we can ship v1.x.x
from the main
branch and v1.x.x-experimental
from the experimental
branch so that users can choose to import.
Of course, experimental
is unstable and supposed to have breaking changes between versions.
Since v1.x.x > v1.x.x-experimental
in SemVer2, it is safe to upgrade the go mod
unattended.
ORAS is a tool for working with OCI artifacts, especially ones in the OCI registries. It provides user-friendly interfaces to push and pull artifacts to / from registries as well as options for advanced scenarios.
The objective of ORAS is simple as transferring artifacts from one place to another.
In the conventional client-server model, the operation of downloading artifacts from the remote registries is referred to as pull, and the operation of uploading artifacts from the remote registry is referred to as push.
This model can be generalized by abstracting the client and the server as targets so that pull and push can be viewed as copying from one target to another (see #8). For instances,
Generally, a target is a content-addressable storage (CAS) with tags. All blobs in a CAS are addressed by their descriptors.
To retrieve a blob,
To store a blob,
It is worth noting that a target is not equal to a registry.
Besides plain blobs, it is natural to store directed acyclic graphs (DAGs) in a CAS. Precisely, all blobs are leaf nodes and most manifests are non-leaf nodes.
An artifact is a rooted DAG where its root node is an OCI manifest. Additionally, artifacts can be grouped by an OCI index, which is also a rooted DAG.
Given a node of a DAG in a CAS, it is efficient to find out all its children. Since CASs are usually not enumerable or indexed, it is not possible to find the parent nodes of an arbitrary node. Nevertheless, some CASs choose to implement or partially implement the functionality of parent node finding. For instances, registries with Manifest Referrers API support are CASs with partially implementation where parent node finding is only available for manifest nodes.
With the concepts above, we can formally define that
It is worth noting that extended copy is possible only if the source CAS supports parent node finding. Based on the scenarios, extended copy can have many options such as opting to copy a sub-DAG rooted by a certain node and all its parent nodes of a certain depth with / without their children.
Optionally, node filters or even node modifiers can be attached to a copy process for advanced scenarios.
Related issues:
Hint: A polytree is a DAG.
We have lots of challenges in developing and maintaining ORAS:
oras-go
library should have as less dependencies as possible.
containerd
as we just need a little bit of that project.containerd
related implementation in the examples
folder.I can use oras.Copy
to copy from a remote (Azure container registry) to a memory target but not the other way around, the Copy function never returns.
Using the same code works when copying to a remote docker registry (running in a docker container, no authentication and reg.PlainHTTP = true
).
As a side note, the development environment is running in docker containers via docker compose.
go 1.17
replace oras.land/oras-go/v2 => github.com/oras-project/oras-go/v2 v2.0.0-20220105111926-3a35b48527fc
Azure container registry
"type": "Microsoft.ContainerRegistry/registries"
"adminUserEnabled": true
Docker Compose version v2.2.3
Client: Docker Engine - Community
Version: 20.10.12
API version: 1.41
Go version: go1.16.12
Git commit: e91ed57
Built: Mon Dec 13 11:45:48 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
Go version: go1.16.12
Git commit: 459d0df
Built: Mon Dec 13 11:43:56 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.12
GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0
var credentials = auth.Credential{Username: "<username>", Password: "<pw>"}
reg, _ := remote.NewRegistry("registry.azurecr.io")
reg.RepositoryOptions.Client = &auth.Client{
Credential: func(c context.Context, s string) (auth.Credential, error) {
return credentials, nil
},
}
src, _ := reg.Repository(ctx, "hello-world")
// No problems here, Copy command returns successfully.
dst := memory.New()
oras.Copy(ctx, src, "latest", dst, "")
// Hang happens trying to copy back to the ACR remote.
dst_copy, _ := reg.Repository(ctx, "hello-world-back")
oras.Copy(ctx, dst, "latest", dst_copy, "")
registry.azurecr.io/hello-world:latest
in present in the ACR with the following manifest:
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1469,
"digest": "sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 2479,
"digest": "sha256:2db29710123e3e53a794f2694094b9b4338aa9ee5c40b930cb8063a1be392c54"
}
]
}
The current PR doesn't implement pagination and I would rather checkin the base implementation and add this incrementally.
We are prototyping the use of oras-go in a kubernetes controller. The controller runs in a pod with a sourcecontext
securityContext:
readOnlyRootFilesystem: true
When trying to log to the remote registry, we get the error:
mkdir /.docker: read-only file system
The happens when login with
authorizerLoginOpts := []auth.LoginOption{
auth.WithLoginContext(ctx(c.out, c.debug)),
auth.WithLoginHostname(host),
auth.WithLoginUsername(username),
auth.WithLoginSecret(password),
auth.WithLoginUserAgent("****"),
}
if operation.insecure {
authorizerLoginOpts = append(authorizerLoginOpts, auth.WithLoginInsecure())
}
if err := c.authorizer.LoginWithOpts(authorizerLoginOpts...); err != nil {
return fmt.Errorf("login failed: %s", err)
}
Heyo! I have a very naive question about the WriterOpts here:
Line 221 in 9ee1349
How can there be a Desc attribute if "Desc" isn't part of the structure here?
Line 24 in 9ee1349
Is it somehow added with the loop to update the opts, or is the field supposed to be the InputHash or OutputHash?
Thanks!
Once Draft.2 of the ORAS Artifact manifest is released, implement the changes in oras-go, then update oras
Since oras is not just a tool for transfering contents between storages, it is also a fully functional registry client now. Therefore, we need to design a proper user-friendly interface for the Registry.
Based on Docker Registry HTTP API V2, the registry client should have the following abilities:
Since ability 2 - 5 are already covered by the Target
interface, the registry interface should just need two more methods:
We can open another issue for the referrers API support
Once we have the least code working, we should update the examples so that people can try themselves.
Examples (originally from @sajayantony 's list):
Failed to add package oras.land/oras-go/v2
to a project.
$ go get oras.land/oras-go/v2
go: downloading oras.land/oras-go v1.1.0
go get: module oras.land/oras-go@upgrade found (v1.1.0), but does not contain package oras.land/oras-go/v2
If the entry is manually added to go.mod
, it reports error after running go mod tidy
$ go mod tidy
go: errors parsing go.mod:
D:\Source\Repos\notation\go.mod:13:2: unrecognized import path "oras.land/oras-go/v2": reading https://oras.land/oras-go/v2?go-get=1: 404 Not Found
Workaround: Add the following entry to the go.mod
file:
require oras.land/oras-go/v2 v2.0.0
replace oras.land/oras-go/v2 => github.com/oras-project/oras-go/v2 v2.0.0-20220124135338-24c5637bde7b
PUT
requests are attempted twice. The body is read for auth and gets empty for the retried requests.
A common scenario is to get content such as manifest and blob into the memory for further processing.
Although the Resolve()
and Fetch()
methods in the Target
interface provides a simple clean secure way to read content, we still need a simplier UX such as GetManifest()
and GetBlob()
(or can be further combined as GetContent()
). Similarly, we might also need PutContent()
in the oras
package.
In v1
, logs are traced via github.com/sirupsen/logrus as containerd
uses it.
Since not all adopters use logrus
or go built-in log
, we might need something similar to net/http/httptrace.
Hello,
I've been following the project readme as listed but it seems that the readme is severely out of date and many functions are either completely changed or removed altogether. This is the link I've been following: https://oras.land/client_libraries/go/
Where can I find a readme that would work with the latest release?
Is it possible to list images and tags from the remote repository?
EDIT: To give some more context.
I'm writing a CLI tool that supports plugins and I'm using oras-go
to publish and install plugins to and from a docker repo where each plugin is a tag I'd like to be able to provide a list or search command that allows discovering plugins and display some information from it's annotations.
I've been looking around the fetcher
but it seems to require that you provide a reference with a tag.
distribution supports manifest conversion between manifest types by setting the Accept
header.
There are two types of conversion supported by distribution
as in the code
Depending on the registry implementation, more conversions are supported. For instance, ACR supports
While having the convenience of convertion, we have issues with digests and applications that depend on digests (e.g. signatures) since the converted manifest cannot be accessed by digest.
Output of a converted docker schema2 manifest:
$ curl "http://localhost:5000/v2/alpine/manifests/latest" -I
HTTP/1.1 200 OK
Content-Length: 2735
Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
Docker-Content-Digest: sha256:c2daf493729aa76b0e4ac53e56d3e45889d17aa94773593d18d7f6924f0c7e1b
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:c2daf493729aa76b0e4ac53e56d3e45889d17aa94773593d18d7f6924f0c7e1b"
X-Content-Type-Options: nosniff
Date: Tue, 07 Dec 2021 06:17:47 GMT
$ curl "http://localhost:5000/v2/alpine/manifests/sha256:c2daf493729aa76b0e4ac53e56d3e45889d17aa94773593d18d7f6924f0c7e1b" -I
HTTP/1.1 404 Not Found
Content-Type: application/json
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Tue, 07 Dec 2021 06:17:56 GMT
Content-Length: 182
To avoid ambiguity, oras-go
is designed not to do type conversion.
#52 covers the basic copy scenarios. However, people may want extra flexibilities on oras
. In that case, we can add ...Option
support just like what we have now.
To make Copy
useful, we need to implement a OCI
target based on the current oras.land/oras-go/pkg/content.OCI
.
OCI Layout specification can be found at https://github.com/opencontainers/image-spec/blob/main/image-layout.md
First thing first, thank ORAS maintainers for separating the oras
utility into oras
CLI and oras-go
library.
Since oras-go
is a Go library and not a command line tool any more, the standard project layout is not proper for this repository anymore.
Consider that the following import block
import (
"oras.land/oras-go"
"oras.land/oras-go/content"
)
will be better than
import (
"oras.land/oras-go/pkg/oras"
"oras.land/oras-go/pkg/content"
)
In the future, when oras
reaches v1.0.0
or even v2.0.0
, it is better to have
import (
"oras.land/oras-go/v2"
"oras.land/oras-go/v2/content"
)
than
import (
"oras.land/oras-go/v2/pkg/oras"
"oras.land/oras-go/v2/pkg/content"
)
pkg/oras
to the root folder.pkg
to the root folder so that there will be no more pkg
folder.Since we are decoupling the containerd
while trying to introduce minimum backward compatibility issues, having a new project layout allows us to have new packages and the old ones at the same time. Then we can mark the old ones deprecated and slowly move them out.
All comments and suggestion are welcome! ๐
We need to have a roadmap on what will be there for version 0.6.0
or even 1.0.0
. With out a roadmap, we will lose our focus.
Making oras-go
stable is more and more important as it is getting more and more adopters.
Related issues:
The library requires a way to authenticate remote stores.
According to distribution, pushing images requires pull,push
actions. However, our registry client only has hint for push
.
We need to update Makefile and scripts once we revamp the oras-go
library.
Based on OCI call discussion today, we should remove +json as a default
Since we have oras-go
version 1 released, we have to update the go.mod
in the main
branch to oras.land/oras-go/v2
as documented in the go doc.
If the module is released at major version 2 or higher, the module path must end with a major version suffix like
/v2
. This may or may not be part of the subdirectory name. For example, the module with pathgolang.org/x/repo/sub/v2
could be in the/sub
or/sub/v2
subdirectory of the repositorygolang.org/x/repo
.
Optionally, we can have an adaptor example in the example
folder to adapt remotes.Resolver
to our Target
interface if users insist to use containerd
implementation. Since the example
folder is a nested module, it will not impact the dependencies of the main module.
The current copy implementation moves a source artifact to a target. The library needs to support moving references to support the following options
artifactType=sbom/example
Copy with specific references, included
oras cp registry.wabbit-networks.io/net-monitor:v1 \
registry.acme-rockets.io/library/net-monitor:v1 \
-r \
-m application/vnd.cncf.notary.v2 \
-m sbom/example
Copy with specific references, excluded
oras cp registry.wabbit-networks.io/net-monitor:v1 \
--destination registry.acme-rockets.io/library/net-monitor:v1 \
-r \
-x sbom/example
Copy with specific references, included, filtered by annotation (inclusion)
oras cp registry.wabbit-networks.io/net-monitor:v1 \
-r \
-m application/vnd.cncf.notary.v2 \
--match-annotation "org.cncf.notary.v2.signature.subject=acme-rockets.io"
Copy with specific references, included, filtered by annotation (exclusion)
oras cp registry.wabbit-networks.io/net-monitor:v1 \
-r \
-m application/vnd.cncf.notary.v2 \
--exclude-annotation "org.cncf.notary.v2.signature.subject=wabbit-networks.io"
Once the new implementation is done and well tested, we can drop the old implementaion in pkg
. For users who want the old implementation, they can pin the oras-go
version to v0.5.0
.
To make Copy
useful, we need to implement a registry client for the Registry
target.
To make Copy
useful, we need to implement a File
target based on the current oras.land/oras-go/pkg/content.File
.
A file target consists of
In the IndexAll()
function, the done
channel is not actually used. We should improve the code.
See https://github.com/oras-project/oras-go/blob/main/internal/graph/memory.go#L60
Targeting releasing v2.0.0
, it is important to stabilize the oras-go
library with the following criteria since oras-go
is and will be used as a fundamental building block of production projects and even security products:
This issue is used to track the process of the oras-go
revamp in order to make oras-go
meeting the above releasing criteria. Tasks are created as follows:
/cc @sajayantony @SteveLasker @jdolitsky @deitch @northtyphoon
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.