cse-labs / ratify Goto Github PK

This documentation would provide guidance for plugin developers

• referrer store gRPC plugin dev docs
• verifier gRPC plugin dev docs

Unit tests with coverage of equivalent state for the updated VerifySubject function of the Executor

Related to #6

Description:
Verify implementation of BYOK with CRDs
Ratify config updates with CRDs needs better documentation in upstream

AC:

• User understands how to use CRDs with bring your own public key using docs
• Take conclusion to Ratify community call

Description:
A failed verification is treated as if an exception is being thrown.
Policy provider in JSON config - Passthrough mode helps to verify robust policies in rego.

Without passthrough mode expectation is that if validation of subject fails, loop breaks out.
With passthrough mode on, validation failure or exception thrown, loop continues on to next artifact.

Acceptance Criteria:

• Verify behavior of execution loop for following scenarios:
  • execution mode is passthrough
• Discuss with team if current behavior is valid/acceptable
• Documentation

Description:
Currently, constraint templates exist in library folder.

Acceptance Criteria:

• Document current constraints
• Goal is to write constraints that meet criteria
  • Each image must be signed
  • Each image must have an SBOM attached. The SBOM must be signed.
  • Each image must have a vulnerability report attached. The report must be signed
• PR into scenario repo

In current state, policy providers in Ratify serve two purposes:

1. To determine whether a reference artifact needs to be verified (this is always true)
2. To make a final quality judgment as to whether the subject passes verification or not (when mode is passthrough, this is always true)

The introduction of passthrough mode allows something external to Ratify make that call -- like Gatekeeper in a Kubernetes cluster. In instances where no external entity is available to step in and make the final decision, the current "any" or "all" checks are rudimentary. Ideally, an orchestrator could be configured to evaluate rego constraints like Gatekeeper through the inclusion of the OPA Constraint Framework library.

The addition of this function may require elevating "constraint evaluation" to be another type of plugin -- just for orchestrators instead the Executor.

Currently it's possible to jam in nonsense, label it as SPDX, and the SBOM verifier is happy to accept it. We want to validate that artifacts are in the correct format before returning isSuccess: true

This is a spike to prove out the idea. We want to start small with a single example/type versus trying to solve the general problem for all possible use cases

Acceptance Criteria

• New verifier plugin that uses JSON-Schema to validate documents
• Uses a hardcoded schema for application/spdx+json
  • The idea is that in the future, you might specify schema via config, but don't dump a lot of time on that - keep it simple

#58 (comment)

The bulk of work surrounding CRDs for the plugin configuration is being handled in the upstream repository. As we shift configuration management out of the Executor and into the purview of the orchestrators, we need to make sure that the orchestrator intended to serve as the Gatekeeper external data provider is configured to be listening for changes to the cache of the CRDs and appropriately add/update/remove such plugin from the Executor with the functions added with #7

gRPC verifier plugins need to be able to call the store functions the Executor exposes as a part of #5. Since the Executor is a part of the core package, it cannot expose service endpoints. This is the responsibility of the orchestrator component.

• update the Gatekeeper external data provider orchestrator to implement the required functions as the server. They should call the appropriate Executor functions.
• unit tests

This includes but is not limited to:

• details on the orchestrators shipped with Ratify (CLI & http server for Gatekeeper external data)
• components not formally plugins
  • configuration (JSON file, CRD)
  • policy provider
• gRPC plugins can only be supported by server-based orchestrators (#8 will produce required proto files for implementation)
• ExecutionMode what are the options, what impact does it have...

Description
Original findings - Current findings reported here

Current behaviour:
If verifier cannot verify - execution moves to next verifier
If verifier can verify - execution loop exits
Assumption is that per artifact type there is only one verifier

Expectation:
If verifier cannot verify - execution moves to next verifier
If verifier can verify - execution moves to next verifier

Acceptance Criteria

• Update and test code for expected behavior listed above
• Understand impact of this change
• Explore nested artifacts and behavior
• Share finding with team
• Document current behavior
• Understand legitimate scenarios which may require this expected behavior

Description

How does Ratify (Oras auth providers) handle authenticated connectivity to single/multiple container registries.

How/can this be achieved?

For example, assume both of these registries require auth, what happens when the following is run:

kubectl run demo --image=foobar1.azurecr.io/test/helloworld:signed
kubectl run demo --image=foobar2.azurecr.io/test/helloworld:signed

Acceptance Criteria

A quick demo and or explanation of how auth currently works within Ratify (Oras auth providers).

This is documentation for comprehension/understanding and not for integration/development per se.

This would include but is not limited to:

• method signatures
• description of the logic of the loop over artifacts
• meaningful diagrams which add clarity
• keep plugin integration discussion at a higher level but linking to the plugin dev docs is imperative

We originally had a vision of a long-running branch here to stage significant changes. We're now currently aiming at pushing small incremental updates to upstream

It would remove a bit of friction for main in both branches to be in sync, but we currently have a few diffs on the cse side

Acceptance Criteria

• cse-labs main & deislabs main are in sync
• Changes that we want to keep are PR'd into deislabs
• Changes that are just working notes consolidated into a long-running notes branch in the cse-labs repo
• Team knows where to stage & target different types of PR's

• Remove unused code
• Ensure all tests pass

For the file watching capability in the config manager, if the file hashes don't match, the ManagePlugins function added with #7

Any orchestrator using the json configuration file will be impacted by these changes.

Description:
Define the purpose of verifiers in current state. What would the purpose of verifiers be in desired state? What should the boundary between responsibility of verifier vs. responsibility of constraint/policy be? This should probably be done by all groups after there is greater familiarity with the codebase and the different components. Ideally, the whole group would reconcile and agree upon the definitions/boundary of desired state.

AC

• Document proposal of design
• Design discussion with team to reach a shared understanding and consensus
• Reviewing community call recording from Nov 15th - Ratify Community Call 2022 Nov 15 - YouTube

Description
Using Plugins from ACR

AC

• Push a plugin to ACR
• Update Ratify to pull images via ORAS
  • At startup when using config.json
  • When registering a new plugin using CRD

• *.proto file containing the request/response messages
• *.proto file containing the required service definition

Currently, in pkg/verifier/plugin/plugin.go, plugins perform the logic to traverse nestedReferences. Other verifiers, such as notaryv2 do not include this logic, and it is confusing to determine how this feature works

The goal of this story is to move that control flow out of plugins and into pkg/executor/core/executor.go so that all plugins execute in a consistent manner

Note: this item was already included in the CSE<->Ratify design document

Note 2: Noel has some PoC code on this topic

** Acceptance Criteria **

• Executor handles nestedReferences and does the work to traverse down into the tree
• Do not pass in an Executor to the Plugin verifier
• Move config into executor block for nested references (as discussed with upstream team, the config will stay with the verifiers for now)
• All tests pass

#56 (comment)

As a ratify developer, I want to provide extensibility for verifier plugins by providing a gRPC option so that ratify plugin developers can bring their own gRPC verifier plugins

• a *.proto file containing the relevant messages required for verifier plugins
• a *.proto file containing the verifier plugin service definition
• a verifier client factory (Add, Update, Delete, Get). Expectation is the factory is similar to the client provider for the k8s secrets store csi driver. However, since there is a requirement to add/update/delete plugins individually, we can't rely on a single Get function as the csi driver does.
• unit tests for the factory
• A wrapper/runner which actually uses the gRPC client to call the verifier server. This should expose similar functionality as done binary plugins here.
• unit tests for execution loop

• Remove unused code
  • binary runner
  • Notation verifier plugin
  • cosign verifier plugin
  • license checker verifier plugin
  • SBOM verifier plugin
  • sample verifier plugin
• Ensure all tests pass

this is the http server which exposes a single endpoint spec'd to integrate with Gatekeeper as an external data provider

• update to reference v2 Executor
• update to call v2 VerifySubject function

this is the http server which exposes a single endpoint spec'd to integrate with Gatekeeper as an external data provider

• Remove unused code
• Ensure all tests pass

Reasonable coverage of functions added to the Executor

• GetBlob
• GetReferenceManifest

Related to #5

Unit tests with coverage equivalent to existing state

• store plugin runner
• verifier plugin runner

• update to reference v2 Executor
• update to call v2 VerifySubject function

Coverage of the new plugin management functions

• AddPlugin
• UpdatePlugin
• RemovePlugin
• ManagePlugins

Related to #7

As a ratify developer, I want to provide extensibility for store plugins by providing a gRPC option so that ratify plugin developers can bring their own gRPC store plugins

• a *.proto file containing the relevant messages required for store plugins
• a *.proto file containing the store plugin service definition
• a store client factory (Add, Update, Delete, Get). Expectation is the factory is similar to the client provider for the k8s secrets store csi driver. However, since there is a requirement to add/update/delete plugins individually, we can't rely on a single Get function as the csi driver does.
• unit tests for the factory
• A wrapper/runner which actually uses the gRPC client to call the store server. This should expose similar functionality as done binary plugins here.
• unit tests for execution loop

1. Evaluate the current state of plugin documentation from the perspective of a prospective plugin developer
2. Build upon the docs added with #3 to create a cohesive document devoted to plugin development

This should include:

• descriptions of the types of plugins (referrer store, verifier)
• descriptions of the integration paths (binary/subprocess, gRPC)
• specific interfaces & requirements

Currently there is a single Dockerfile for Ratify. It lives in the httpserver folder and the entrypoint command is serve to boot up an http server.

Although ORAS doesn't have a server component / option, it does provide the option to run the CLI in a container. Gatekeeper also has the gator CLI which can be run in a container too.

Gatekeeper also has a Dockerfile for CRDs which builds separately from the core image

What steps did you take and what happened:
The current TestStore (pkg/referrerstore/mocks/types.go) in the codebase is a partial implementation and cannot be customized to handle nested artifact tests. It only allows one hardcoded return for referrers and subject lookup is hardcoded to look at a tag.

What did you expect to happen:
A test object should be easy to configure and use. It should also be robust enough to handle a variety of test scenarios. A new test store (pkg/referrerstore/mocks/memory_store.go) is being added for nested reference tests and it should superceede and replace the other TestStore.

Anything else you would like to add:
The new test store could also benefit from a fluent builder to make test code more concise, readable, and clean from magic values. A proposed example below:

	store := mocks.CreateEmptyMemoryTestStore()

	subjectUri := store.AddSubject("test-image-subject", mocks.DockerMediaType).
		WithReferrer("sbom", mocks.ArtifactMediaType, mocks.SbomArtifactType).
		WithReferrer("image-sig", mocks.ArtifactMediaType, mocks.SignatureArtifactType).
		GetTestUri()

	store.AddSubject("sbom", mocks.ArtifactMediaType).
		WithReferrer("sbom-sig", mocks.ArtifactMediaType, mocks.SignatureArtifactType)

Environment:

• Ratify version: (use the image tag):
• Kubernetes version: (use kubectl version):

Whether e2e or integration, tests need to be included to ensure confidence in the gRPC plugin path through tangible results.

The requirement isn't 1 standalone test for each, but each should be covered by at least 1 test:

• happy path for stores
  • GetBlobContent
  • GetReferenceManifest
  • GetSubjectDescriptor
  • ListReferrers
    • with 1+ referrer artifacts returned
    • with 0 referrer artifacts returned
• exception/error path for stores
  • GetBlobContent
  • GetReferenceManifest
  • GetSubjectDescriptor
  • ListReferrers

Description
Currently 2 SBOM plugins - SBOM verifier, License checker
Need to check efficiency or capability of SBOMs generated by MS SBOM tool

Acceptance Criteria

• Need an image that has an associated SBOM
• Verify that SBOM verifier works as expected with above SBOM - should always pass
• Verify that license checker always fails. JSON based SPDX SBOMs would fail the license checker.
• Verify with Gatekeeper in a Kubernetes cluster.
• Documenting verified capabilities.
• Documenting any misalignments and creating new Issue if required.
• Share findings with team.

Description
We have some great documentation that came out of the exploration phase.
We need to push the relevant/beneficial ones upstream.

AC

• Look into docs created as part of exploration phase
  • ratify/hack/42-verifiers at 42-verifiers · cse-labs/ratify (github.com)
• Combine relevant/useful docs
  • Possible code snippets
  • Inline script blocks
• Verify with v-team - to see if we are missing anything
• Create PR to upstream
• Bring up discussion in Ratify community call for visibility

We've created an out of tree verifier, but have not done the same with a store. The goal of this issue is to create a simple store and answer the following questions

Acceptance Criteria

• Does an out of tree store work with built-in verifiers?
• Does an out of tree store work with out-of-tree verifiers?
• Describe how this could be used similarly to existing customers
• Demonstrate that we can load out of tree store plugins without rebuilding Ratify/Helm chart (similar to verifiers)
  • ex: via helm post install / kustomize
• Document creating out of tree stores

There is currently a verifier plugin template located at https://github.com/noelbundick-msft/ratify-verifier-plugin

Noel will not maintain this

We need to get it moved into deislabs, if possible

Acceptance Criteria

• Coordinate with Cloud Native PM's
• Validate/update docs to ensure they work with latest Ratify
• Merged PR for all of that into the new template repo

Description
Need to find out if something like an SBOM crawler to scan contents of SBOM exists.

AC
New AC below

Create prototype that will

• Parse VEX JSON format
• Check if each entry has published CVEs
• Check existing open source tools to see how functionality is implemented - Trivy

Unit tests for the revamped verify function with coverage equivalent to current state

related to #4

This is expected to be in the same vein as the existing VerifySubject function. It will be unused code and a separate issue will handle the switch over to this function from the current.

Additional params:

• store plugin configs
• verifier plugin configs

Although the respective config objects should be the parameters, when it comes to execution, this function should accept only the "name" property to be provided. If only the name is provided, then the plugin in question must exist in the cached plugin instantiations.

• unit tests

The devcontainer currently does some useful things to put the default wabbitnetworks.azurecr.io cert into the correct location

With the inclusion of notation rc1 changes, the cert location has changed from ~/.ratify/ratify-certs/notary to ~/.ratify/ratify-certs/notary/truststore/x509/ca/certs. We need to land the file in the correct location when we set it up in .devcontainer/post-create.sh

We also need to merge #53

Acceptance Criteria

• Verify current state of devcontainer and any work that needs to get it to a good state
• devcontainer is updated to "just work" with the sample image. Devs should be able to create a new codespace, hit F5, and successfully verify wabbitnetworks.azurecr.io/test/net-monitor:signed
  • Note, as of 2022-12-05, the sample image is currently built using ORAS artifacts and doesn't work with current Ratify. That work is tracked here: ratify-project#465
  • Add ORAS to devcontainer

Description:
This entails exploring what is currently possible with out of tree plugins, what are the limitations.
Out of Tree plugin -- one that is not included in the ratify repo.

Acceptance Criteria:
Try working with out of tree plugins with following ways:

• Rebuilding around Ratify. Build should include pulling in docker file for your plugin or collection of plugins. Idea is you own the image.
• Modified Helm chart so plugin can be synced and mounted as a volume. Idea is you own the helm chart.
• While running Ratify as a binary/CLI, use plugin path in Ratify config.
• Make sure execution loop is called - by calling ORAS store and verifying that store can be executed etc.
• Create a basic custom plugin that calls ORAS store
• Share findings with team.
• Documenting verified capabilities. Documenting any misalignments and creating new Issue if required.

Just tested out changing helm values and running helm upgrade and seems to work fine. Interested to hear if anyone is still having issues with this. - Josh (not sure why it is not allowing me to add my own comment so it looks like Noel wrote this but he did not)

• Remove unused code
• Ensure all tests pass

Passthrough mode used to be important & impacted control flow. Now, it only forces the top-level isSuccess property to true

It may not be needed anymore. Figure it out, make a recommendation in Ratify issues + community call (or close this)

This involves updating the actual plugins themselves. The specifics of the updated Verify signature are dependent on decisions made as a part of #4.

Verifier plugins to update include:

• Notation (Notary v2)
• cosign
• license checker
• SBOM
• Sample
• Unit tests

For performance reasons, the Executor must be able to instantiate plugins and hold them in memory separate from core VerifySubject request.

A v2 of the Executor struct should be made & the Config object should be removed. The ReferrerStores and Verifiers can stay -- these will continue to serve as the in memory storage for instantiated plugins.

The following functions must be added to the Executor:

• AddPlugin(type, config)
• UpdatePlugin(type, config)
• RemovePlugin(type, name)
• ManagePlugins(storeConfigs, verifierConfigs) the Executor shall handle the appropriate add/update/remove in bulk
• unit test

Whether e2e or integration, tests need to be included to ensure confidence in the gRPC plugin path through tangible results.

The requirement isn't 1 standalone test for each, but each should be covered by at least 1 test:

• happy path for verifiers
• exception/error path for verifiers
• failure but not exception for verifiers (like the reference artifact is a signature, but it cannot be verified)

To decouple the verifiers and stores, the Executor must have public/accessible functions which can call the appropriate store and return the result.

The parameters must include:

• the configured name of the store plugin that should be called (as it has been registered with the Executor)
• the data that the store plugin is expecting for each respective function

Required functions:

1. GetReferenceManifest
2. GetBlobContent

• Unit tests

TODO: May need to rephrase to provide more context/clarity

Decouple store context from binary plugins, figure out how orchestration works so that verify is NOT interacting w/ store directly -- provides flexibility for verifier interface

This is expected to be introduced alongside the existing Verify function here in the plugin runner for binary verifiers. It will be unused code and a separate issue will handle the switch over to this function from the current.

In place of accepting the store config as a parameter, the verifier should be passed the information required to call the Executor. The functions the Executor will exposed are handled with issue #5

Includes unit tests