Design
According to Enclave Confidential Containers (Enclave-CC) Design and Architecture, shim-rune should launch the agent enclave container when pod creation. The agent enclave container is deployed in the form of OCI bundle rather than a container image, and thus shim-rune can call rune to start it. The agent enclave container has the same life cycle as the pod. Its main job is to receive and then load the actual workload's container image into an app enclave.
As is shown in the picture, the binary name of shim-rune is containerd-shim-rune-v2. shim-rune should implement the Containerd Runtime V2 (Shim API) for Enclave-CC. With shim-rune, Kubernetes can launch Pod and OCI-compatible containers with one shim per Pod.
Goals
- implements the Containerd Runtime V2 (Shim API)
- use
rune OCI runtime to manage container
- launch agent enclave container during PodSandbox creation
- complete PullIImage with agent enclave container with ttrpc communication
- launch application enclave container with FUSE encryption filesystem generated by agent enclave container.
- kill/cleanup agent enclave container resource when stopping PodSandbox
Workflow
1. RunPodSandbox
Compared with containerd-shim-runc-v2, shim-rune should add the following functions when launching the pod sandbox.
- In the
Create API of shim-rune, shim-rune will create an agent enclave container with the pre-created oci bundle after the pause container is created successfully.
- In the
Start API of shim-rune, shim-rune will start the agent enclave container after the pause container is started successfully.
A pre-created oci bundle for the agent enclave container is needed instead of the form of a container image because the agent enclave container is the first component along with pod creation and there is no direct support for it to pull and unpack it in a TEE. The agent_container_instance field in the configuration file of shim-rune(In the Appendix section) shows the host path of oci bundle for the agent enclave container.
The agent enclave container has the same life cycle as the pod. Its main job is to receive and then load the actual workload's container image into an app enclave. The contents of the bundle file are as follows:
tree /opt/enclave-cc/agent-instance/ -L 1
/opt/enclave-cc/agent-instance/
├── config.json
└── rootfs
Note that oci bundle of the agent enclave container (agent_container_instance field in the shim-rune configuration file) must be read-only because the agent enclave containers of all Pods on this node share this bundle. Once modified, it will affect the behavior of existing or newly created agent enclave containers. Therefore, every time the agent enclave container is created, agent_container_instance is used as the read-only layer and then overlays read and write layers to aggregate the final oci bundle for the agent enclave container.
In addition, the oci bundle of agent container may have multiple different versions, they contain different versions of untrusted PAL (even support different LibOS) and its dependencies, such as configuration files. shim-rune just needs to call rune and pass the location of its oci bundle to rune.
rune receives the path of the oci bundle passed by shim-rune, parses the oci container configuration file, creates and enters a new mount namespace, and uses bind mount to mount the rootfs directory tree in the oci bundle as the container rootfs. Finally, LibOS is loaded and initialized through the PAL API pal_init() as the app container's No. 1 process runelet. Please refer to rune for the detail information.
2. Pull application Image
- The
PullImage API of shim-rune will send PullImageReq to the agent enclave container.
- The agent enclave program uses the
image-rs to pull images and uses the encrypted file system capabilities provided by LibOS to create an encrypted unionfs (based sefs) image.
- The agent enclave store the encrypted unionfs (based sefs) image in host. We will discuss the path to store encrypted unionfs (based sefs) image in the next subsection(Create application container section).
Communication protocol
Below is the Image communication protocol between the shim-rune and agent enclave container. The communication protocol is referred to kata.
syntax = "proto3";
...
// Image defines the public APIs for managing images.
service Image {
// PullImage pulls an image with authentication config.
rpc PullImage(PullImageRequest) returns (PullImageResponse) {}
}
message PullImageRequest {
// Image name (e.g. docker.io/library/busybox:latest).
string image = 1;
// Unique image identifier, used to avoid duplication when unpacking the image layers.
string container_id = 2;
// Use USERNAME[:PASSWORD] for accessing the registry
string source_creds = 3;
}
message PullImageResponse {
// Reference to the image in use. For most runtimes, this should be an
// image ID or digest.
string image_ref = 1;
}3. Create app container
Ideally, just like kata-agent, the agent enclave container generates the bundle (config.json + rootfs) of the app container based on occlum or gramine. In this way, shim-rune can avoid generating the app container bundle which depends on specific Libos (occlum or gramine).
But LibOS has limited capabilities such as:
- For UnionFS supported by Occlum, the lower layer must be a single-layer FUSE encrypted file system, and the app container image to be aggregated by agent enclave may contain multiple layers, that is, the lower layer is not a single layer.
- Gramine may not be able to easily support the ability to aggregate filesystems.
Possible solutions:
- In the long run, promote LibOS to realize the aggregation capability of supporting multiple lower layers.
- In the short to medium term, the enclave-agent program cannot rely on the rootfs aggregation capability of a specific LibOS to construct the oci bundle of the app container. shim-rune is responsible for generating the bundle（config.json + rootfs）of the app container.
Considering these limitations and after discussing with Libos team, the first stage enclave-cc uses the following methods to run the application enclave container.
Occlum
Refer to the occlum guide of Runtime boot pre-generated UnionFS image. A pre-created oci bundle for the boot instance is needed in the host. The boot instance is responsible for using the customized init, mount, and boot a pre-generated UnionFS image.
Considering the implementation of occlum dynamic mount image, shim-rune should overlay the boot instance bundle and encrypted unionfs (based sefs) image to generate the final bundle of the application container. For example:
mount -t overlay overlay -o lowerdir=<path of boot instance bundle>:<path of encrypted unionfs (based sefs) image>, \
upperdir=upper,workdir=work merged
Then the question is where is the location of boot instance bundle and encrypted unionfs (based sefs) image?
- Location of
boot instance bundle
One host only needs a pre-created occlum oci bundle for the boot instance. The boot_container_instance field in shim configure file can show the location.
The pre-generated boot template instance looks like an oci bundle rootfs on the host side. It contains the occlum boot template instance.
tree /opt/enclave-cc/boot-instance -L 1
/opt/enclave-cc/boot-instance
└── rootfs
1 directory, 1 file
- Location of
encrypted unionfs (based sefs) image
One pod only has one agent enclave container. The agent container needs to manage multiple images. After the agent container pulls an image, it will convert all layers about this image into an encrypted unionfs (based sefs) image in the host.
Since one image corresponds to a encrypted unionfs (based sefs) image, we can refer to the implement in kata-agent, agent enclave contianer can generate based on image name. The agent enclave contianer can store the encrypted unionfs (based sefs) image to directory distinguished by , such as <agent_container_bundle_path>/rootfs/run/rune/<cid>/rootfs dir.
When shim-rune launch the application enclave container, shim-rune will generate the same based on the image name, and then find the corresponding encrypted unionfs (based sefs) image. Then shim-rune overlay the encrypted unionfs (based sefs) image and boot instance bundle to generate the final app container bundle.
Gramine
TODO
At last, shim-rune call rune to launch the app enclave container. The process is similar to launch an agent enclave container with rune.
4. Stop sandbox
Compared with containerd-shim-runc-v2, shim-rune adds the work of kill/cleanup work for the agent enclave container.
How to integrate with containerd
Because shim-rune supports containerd shim v2 API, you can add the associated configurations for shim-rune in the containerd config file, e.g, /etc/containerd/config.toml, on your system.
[plugins.cri.containerd]
...
[plugins.cri.containerd.runtimes.rune]
runtime_type = "io.containerd.rune.v2"
then restart containerd on your system.
Appendix
shim configuration sample
log_level = "info" # "debug" "info" "warn" "error"
[containerd]
socket = "/run/containerd/containerd.sock"
agent_container_instance = "/opt/enclave-cc/agent-instance/"
boot_container_instance = "/opt/enclave-cc/boot-instance/"
agent_container_root_dir = "/run/containerd/agentenclave"
agent_url = "tcp://0.0.0.0:7788"
where:
- @log_level: specify the log level for shim-rune.
- @socket: the containerd socket
- @agent_container_instance: the host path of pre-created oci bundle for agent enclave container
- @boot_container_instance: the host path of pre-created oci bundle for boot instance
- @agent_container_root_dir: the root dir of agent enclave container running state
- @agent_url: the listening address of the agent container, through which the shim communicates with the agent container to perform PullImage. (support unix and tcp communication)
- Note: Currently occlum 0.27 version does not support cross_world_uds, temporarily implement ttrpc communication based on tcp
- After occlum releases NGO at the end of June, the communication between shim-rune and agent container will be switched to ttrpc based on cross_world_uds
Reference
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent