Comments (10)
mythi
commented on July 16, 2024
@Xynnn007 do you see anything obvious in the logs why the RCAR handshake fails?
from enclave-cc.
Xynnn007
commented on July 16, 2024
@niteeshkd Please check the following
-
Anymore logs on agent-enclave stderr? The last
WARNis as expected. There should be a new log item to show whether the new round handshake passes. -
On the KBS side. Whether
default/security-policy/testexists
from enclave-cc.
niteeshkd
commented on July 16, 2024
Here is the full content of that stderr file.
$ sudo cat /run/containerd/agent-enclave/8c494eedbcb48611120555e9c331017a57b790c226e303209d497b08d99f0a28/stderr
[2023-12-11T18:10:47Z INFO enclave_agent] ttRPC server started: "tcp://127.0.0.1:7788"
[2023-12-11T18:10:47Z INFO enclave_agent::services::images] Pulling "docker.io/niteeshkd/occlum-hello-world:enc"
[2023-12-11T18:10:48Z INFO image_rs::resource::kbs] secure channel uses native-aa
[2023-12-11T18:10:48Z WARN kbs_protocol::client::rcar_client] Authenticating with KBS failed. Perform a new RCAR handshake: ErrorInformation {
error_type: "https://github.com/confidential-containers/kbs/errors/InvalidRequest",
detail: "The request is invalid: parse Authorization header failed: invalid Header provided",
}
[2023-12-11T18:12:00Z INFO enclave_agent::services::images] Pulling "docker.io/niteeshkd/occlum-hello-world:enc"
[2023-12-11T18:12:00Z INFO image_rs::resource::kbs] secure channel uses native-aa
[2023-12-11T18:12:01Z WARN kbs_protocol::client::rcar_client] Authenticating with KBS failed. Perform a new RCAR handshake: ErrorInformation {
error_type: "https://github.com/confidential-containers/kbs/errors/InvalidRequest",
detail: "The request is invalid: parse Authorization header failed: invalid Header provided",
}
[2023-12-11T18:13:26Z INFO enclave_agent::services::images] Pulling "docker.io/niteeshkd/occlum-hello-world:enc"
[2023-12-11T18:13:26Z INFO image_rs::resource::kbs] secure channel uses native-aa
[2023-12-11T18:13:27Z WARN kbs_protocol::client::rcar_client] Authenticating with KBS failed. Perform a new RCAR handshake: ErrorInformation {
error_type: "https://github.com/confidential-containers/kbs/errors/InvalidRequest",
detail: "The request is invalid: parse Authorization header failed: invalid Header provided",
}
[2023-12-11T18:15:15Z INFO enclave_agent::services::images] Pulling "docker.io/niteeshkd/occlum-hello-world:enc"
[2023-12-11T18:15:15Z INFO image_rs::resource::kbs] secure channel uses native-aa
[2023-12-11T18:15:16Z WARN kbs_protocol::client::rcar_client] Authenticating with KBS failed. Perform a new RCAR handshake: ErrorInformation {
error_type: "https://github.com/confidential-containers/kbs/errors/InvalidRequest",
detail: "The request is invalid: parse Authorization header failed: invalid Header provided",
}
[2023-12-11T18:17:37Z INFO enclave_agent::services::images] Pulling "docker.io/niteeshkd/occlum-hello-world:enc"
[2023-12-11T18:17:37Z INFO image_rs::resource::kbs] secure channel uses native-aa
[2023-12-11T18:17:38Z WARN kbs_protocol::client::rcar_client] Authenticating with KBS failed. Perform a new RCAR handshake: ErrorInformation {
error_type: "https://github.com/confidential-containers/kbs/errors/InvalidRequest",
detail: "The request is invalid: parse Authorization header failed: invalid Header provided",
}
[2023-12-11T18:18:37Z ERROR ttrpc::asynchronous::server] method handle /grpc.Image/PullImage got error timed out
On KBS side, default/security-policy/test does exist.
$ cat default/security-policy/test
{
"default": [{"type": "accept"}],
"transports": {
"docker": {
"docker.io/niteeshkd/occlum-hello-world:enc": [
{
"type": "sigstoreSigned",
"keyPath": "kbs:///default/cosign-key/1"
}
]
}
}
}
$ tree
.
+-- default
+-- cosign-key
| +-- 1
+-- image-kek
| +-- 11032d96-dccd-46a3-9244-93644d76745f
| +-- 316e53a7-e746-40b4-a3ee-42eee85aa1fa
| +-- ed72bf17-7b73-42c7-9e50-8b069ad12f5f
+-- security-policy
| +-- test
+-- sigstore-config
+-- test
from enclave-cc.
Xynnn007
commented on July 16, 2024
How do you set up the CoCo-AS? Please paste the logs of the CoCo-AS.
BTW, I am afraid that the file sgx_default_qcnl.conf on the KBS/AS side is not configured correctly.
On my dev environment, I use docker-compose to launch KBS cluster. The sgx_default_qcnl.conf mounted on CoCo-AS should specify a pccs_url that could be routed from the CoCo-AS container.
from enclave-cc.
niteeshkd
commented on July 16, 2024
@Xynnn007 I was getting the following log entries for CoCo-AS. I also setup KBS etc. using docker-compose.
$ docker logs kbs-as-1
[2023-12-11T18:07:50Z INFO grpc_as::server] Listen socket: 0.0.0.0:50004
[2023-12-11T18:07:50Z INFO grpc_as::server] Connect to remote RVPS [http://rvps:50003] (gRPC Mode)
[2023-12-11T18:11:48Z WARN attestation_service::verifier::sgx] tee_qv_get_collateral failed: 0xe019
After correcting pccs_url and using use_secure_cert=false in config/sgx_default_qcntl.conf for CoCo-AS, I notice the following.
$ kubectl describe pod enclave-cc-pod
...
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8m46s default-scheduler Successfully assigned default/enclave-cc-pod to b77r44u11-node
Warning Failed 8m32s kubelet Failed to pull image "docker.io/niteeshkd/occlum-hello-world:enc": ttrpc: closed
Warning Failed 7m51s kubelet Failed to pull image "docker.io/niteeshkd/occlum-hello-world:enc": rpc error: code = DeadlineExceeded desc = timed out connecting to tcp socket 127.0.0.1:7788
Normal Pulling 6m34s (x4 over 8m45s) kubelet Pulling image "docker.io/niteeshkd/occlum-hello-world:enc"
Warning Failed 6m34s (x4 over 8m32s) kubelet Error: ErrImagePull
Warning Failed 6m34s (x2 over 7m25s) kubelet Failed to pull image "docker.io/niteeshkd/occlum-hello-world:enc": Agent error: Dead agent
Warning Failed 6m21s (x6 over 8m32s) kubelet Error: ImagePullBackOff
Normal BackOff 3m44s (x17 over 8m32s) kubelet Back-off pulling image "docker.io/niteeshkd/occlum-hello-world:enc"
$ docker logs kbs-as-1
[2023-12-12T03:54:07Z INFO grpc_as::server] Listen socket: 0.0.0.0:50004
[2023-12-12T03:54:07Z INFO grpc_as::server] Connect to remote RVPS [http://rvps:50003] (gRPC Mode)
$ docker logs kbs-kbs-1
[2023-12-12T03:54:09Z INFO api_server::attestation::coco::grpc] connect to remote AS [http://as:50004]
[2023-12-12T03:54:09Z INFO api_server] Starting HTTP server at [0.0.0.0:8080]
[2023-12-12T03:54:09Z INFO actix_server::builder] starting 192 workers
[2023-12-12T03:54:09Z INFO actix_server::server] Tokio runtime found; starting in existing Tokio runtime
[2023-12-12T03:54:51Z INFO api_server::http::resource] Get pkey from auth header
[2023-12-12T03:54:51Z INFO actix_web::middleware::logger] 172.18.0.1 "GET /kbs/v0/resource/default/security-policy/test HTTP/1.1" 401 173 "-" "attestation-agent-kbs-client/0.1.0" 0.000289
[2023-12-12T03:54:51Z INFO api_server::http::attest] request: Json(Request { version: "0.1.0", tee: Sgx, extra_params: "" })
[2023-12-12T03:54:51Z INFO actix_web::middleware::logger] 172.18.0.1 "POST /kbs/v0/auth HTTP/1.1" 200 74 "-" "attestation-agent-kbs-client/0.1.0" 0.000188
[2023-12-12T03:54:51Z INFO api_server::http::attest] Cookie 2ba92875d8b141edb5965a88a7f6de21 attestation Json(Attestation { tee_pubkey: TeePubKey { kty: "RSA", alg: "RSA1_5", k_mod: "tjOjzsDjpeIfAPd92vsqrSIWmNbUA_bH3uZAGE809YHseosh0-a1D8FBuVbEW5HdAbY_oZB5WFEroPQLvByc1TxEx6NaF67AxeYUI0uDcd2j5XGIf7-mVIfVs5vWxA52L4rDrBU6ZKSGMNDwCwk-fjXTHm8bJ-D3P9tn0bg2coTVWLE8MVKfCtLdyVpa9mdhG4_oLRwer1zKbP_EI0EqdpL_tzc3sTtIWMRZOQu0c1Vi7gsRQXUSpubq1N31VqJQldgQ2OS0a4IoUTtqbznsoz1jK-baG0i5SYB_DYZFfGfWkHkuFcEPZ2135igLXOdY8U2tGYPXAVYaGMmbbYm7wQ", k_exp: "AQAB" }, tee_evidence: "{\"quote\":\"AwACAAAAAAAKAA8Ak5pyM/ecTKmUCg2zlX8GB8IAHtrgODb...
VFXeEhSQXZaVUdPZFJRN2N2cVJYYXFJPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCgA=\"}" })
[2023-12-12T03:54:52Z INFO actix_web::middleware::logger] 172.18.0.1 "POST /kbs/v0/attest HTTP/1.1" 200 1839 "-" "attestation-agent-kbs-client/0.1.0" 0.081156
[2023-12-12T03:54:52Z INFO api_server::http::resource] Cookie 2ba92875d8b141edb5965a88a7f6de21 request to get resource
[2023-12-12T03:54:52Z INFO api_server::http::resource] Get pkey from session.
[2023-12-12T03:54:52Z INFO api_server::http::resource] Resource description: ResourceDesc { repository_name: "default", resource_type: "security-policy", resource_tag: "test" }
[2023-12-12T03:54:52Z INFO actix_web::middleware::logger] 172.18.0.1 "GET /kbs/v0/resource/default/security-policy/test HTTP/1.1" 200 906 "-" "attestation-agent-kbs-client/0.1.0" 0.003024
[2023-12-12T03:54:52Z INFO api_server::http::resource] Cookie 2ba92875d8b141edb5965a88a7f6de21 request to get resource
[2023-12-12T03:54:52Z INFO api_server::http::resource] Get pkey from session.
[2023-12-12T03:54:52Z INFO api_server::http::resource] Resource description: ResourceDesc { repository_name: "default", resource_type: "cosign-key", resource_tag: "1" }
[2023-12-12T03:54:52Z INFO actix_web::middleware::logger] 172.18.0.1 "GET /kbs/v0/resource/default/cosign-key/1 HTTP/1.1" 200 725 "-" "attestation-agent-kbs-client/0.1.0" 0.001132
$ sudo cat /run/containerd/agent-enclave/<cid>/stderr
[2023-12-12T03:54:51Z INFO enclave_agent] ttRPC server started: "tcp://127.0.0.1:7788"
[2023-12-12T03:54:51Z INFO enclave_agent::services::images] Pulling "docker.io/niteeshkd/occlum-hello-world:enc"
[2023-12-12T03:54:51Z INFO image_rs::resource::kbs] secure channel uses native-aa
[2023-12-12T03:54:51Z WARN kbs_protocol::client::rcar_client] Authenticating with KBS failed. Perform a new RCAR handshake: ErrorInformation {
error_type: "https://github.com/confidential-containers/kbs/errors/InvalidRequest",
detail: "The request is invalid: parse Authorization header failed: invalid Header provided",
}
[2023-12-12T03:54:52Z INFO sigstore::cosign::client_builder] Rekor public key not provided. Rekor integration disabled
[2023-12-12T03:54:52Z INFO sigstore::cosign::client_builder] No Fulcio cert has been provided. Fulcio integration disabled
[ERROR] occlum-pal: Failed to enter the enclave to execute a LibOS thread (host tid = 15) with error code 0x1003: Out of TCS. (line 21, file src/ocalls/spawn.c)
[ERROR] occlum-pal: Failed to enter the enclave to execute a LibOS thread (host tid = 16) with error code 0x1003: Out of TCS. (line 21, file src/ocalls/spawn.c)
$ cat default/security-policy/test
{
"default": [{"type": "reject"}],
"transports": {
"docker": {
"docker.io/niteeshkd/occlum-hello-world:enc": [
{
"type": "sigstoreSigned",
"keyPath": "kbs:///default/cosign-key/1"
}
]
}
}
}
from enclave-cc.
Xynnn007
commented on July 16, 2024
@niteeshkd Seems that attestation passes now. The left things are why occlum fails to run
[ERROR] occlum-pal: Failed to enter the enclave to execute a LibOS thread (host tid = 15) with error code 0x1003: Out of TCS. (line 21, file src/ocalls/spawn.c)
[ERROR] occlum-pal: Failed to enter the enclave to execute a LibOS thread (host tid = 16) with error code 0x1003: Out of TCS. (line 21, file src/ocalls/spawn.c)
Any ideas? @mythi
from enclave-cc.
mythi
commented on July 16, 2024
We limit max_num_of_threads to 8. In this case, seems that is not enough. Image pull is limited to 3 layers (=threads) at a time and that image does not seem to have even that so something else tries to create those extra threads the libOS cannot satisfy?
@niteeshkd are you still on v0.8.0 tag?
from enclave-cc.
niteeshkd
commented on July 16, 2024
@mythi I used v0.8.0 tag (i.e. kubectl apply -k "github.com/confidential-containers/operator/config/release?ref=v0.8.0" ) and also default (i.e. .../github/operator$ kubectl apply -k config/default/ ) for installing the operator. I noticed the same error.
I also tried with higher values of max_num_of_threads in /opt/confidential-containers/share/enclave-cc-agent-instance/config.json . It did not help.
from enclave-cc.
mythi
commented on July 16, 2024
I also tried with higher values of max_num_of_threads in /opt/confidential-containers/share/enclave-cc-agent-instance/config.json . It did not help.
Unfortunately this is a build time setting and cannot be modified without re-creating the enclave. To rule out the possibility that your image (and getting it prepared) has errors, would it be possible to try docs/ci-setup.md for the CI image we are using?
from enclave-cc.
mythi
commented on July 16, 2024
closing per suggestion from @niteeshkd
from enclave-cc.
Related Issues (20)
- enable dependabot updates for rust dependencies HOT 1
- image pull failures with multi-layer images
- Get rid of eaa-kbc & verdictd HOT 4
- update to combined image-rs+ocicrypt+AA repo HOT 1
- Update Quickstart for v0.8 HOT 2
- CI failed because of key not found HOT 4
- enable signature checks for sample_kbc in CI
- update operator flows for NFD and Debug
- Operator tests are currently failing
- specify rust version used in builds
- Replace apt yum repos for installing Kubernetes HOT 1
- RFC: enclave-cc improvement ideas HOT 4
- Failed to create agent enclave - Invalid enclave metadata. (line 152, file src/pal_enclave.c) HOT 27
- [ERROR] occlum-pal: Failed to create enclave with error code 0x2006: Invalid SGX device. Please make sure SGX module is enabled in the BIOS, and install SGX driver afterwards. (line 152, file src/pal_enclave.c) HOT 5
- shim: RPC protos and dependencies need rework
- Attestation: Verifier evaluate failed: SGX Verifier: REPORT_DATA is different from that in SGX Quote HOT 24
- setup cargo workspace
- deploy enclave-cc failed HOT 5
- how to configure image decryption keys HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from enclave-cc.