Issue with kubernetes default ca.crt certificate when launching cf-extplugtest-server in k8s cluster mode about californium HOT 23 CLOSED

here's the full ca.crt content:

-----BEGIN CERTIFICATE-----
MIC/jCCAeagAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTIzMDYxMzE1MjUzOFoXDTMzMDYxMDE1MjUzOFowFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKoi
2BNY3Kki5Qpr5kXORAOarDstCjsUrEgKocHnuz2U/FOdSCf4fZenIzm0XRchU90A
vVqCBYXdek6ErwhnEm2W6AW9a/Bpv9Gn9urRLDN2F59g4rWtpvwJsX6hy1jUkHME
QXKZWnzmZtEhAheaWIXsOzEude5elEf0j1/d1aMUyGLL6W0GxWQ7MCMKRNN8voi3
205x2E4F8WAJ6j6l1rcZebhYhZyQCbpte34smpu7H4rpg9zVbv+hrG636EdxWnY7
9fLqO5ckCPIxrBJ57XSE2ju3oMxSFataZisEVamaOvINdh+ilz/fRk1rGtW8RRNJ
8EAr+s6TCgKIeUovwX0CAwEAAaNZMFcwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wHQYDVR0OBBYEFD6PYUvADcFpvt0gWsgVC2cz0++hMBUGA1UdEQQO
MAyCCmt1YmVybmV0ZXMwDQYJKoZIhvcNAQELKBQADggEBAFROB/kbkq0vaHfOEGNX
Afxflg7lVOP7UbSMkjBpgf7/2qFN0iCkvKJnFqXm8vcXIs6L/yrKxz4bXep0Yqlg
eXTkk20qFZkuMJYWWtj04phSLCbRiClZ0Yl+vvcxsP+2vVqW0UrRy2B63umVorZB
j0MnNniCO2kq7mADU0c58S+mEQ2zVnfraFokw5x2IFI6hWBvKO7WB/X2g/C7sRl2
FDBHRN2PjgLY824bSNT6v/ENkFVbhhRYt1Btgry0D67DfisgCGiR9RxhZ1vMiWOw1
jbDuUNY26oRF+61qVPRHGLaErbg94yenvDh+cqpO/jF0eH46Z1YH7Hz25gj9Yp+W
q3E=
-----END CERTIFICATE-----

from californium.

Yes, and I see the bug!
A single certificate may be either interpreted as a "chain of one" or a "trust store of one".
Thanks for reporting! I will fix it.

from californium.

So, does the k8s CID cluster work on EKS?

Our setup is 2 kubernetes pods with AWS NLB on top of it and K8S Load balancer service.
With limited testing it appears to be working - our client makes a single handshake, gets connection ID and then sends multiple requests in a short period of time (0 - 1 sec). All of these requests go to a single pod. But different handshakes / connection ids get routed to different pods.

from californium.

no trusted x509 certificates found in 'file:///var/run/secrets/kubernetes.io/serviceaccount/ca.crt'! java.lang.IllegalArgumentException: no trusted x509 certificates found in 'file:///var/run/secrets/kubernetes.io/serviceaccount/ca.crt'

Did you try to login that container and check, if the file is there? If so, what's the content.
My tests with k8s are about 2 years ago, I guess k8s has been moving forward and there maybe now a different way to access the trusted certificates.

from californium.

I copied the ca.crt file and tried to launch SslContextUtilTrustTest suite with TRUST_PEM_LOCATION pointing to this ca.crt file. testLoadPemTrustManager and testLoadPemTrustedCertificates failed.

But you didn't provide copy here? Usually trusted certificates are no data privacy issue, you mainly need to protect the used ones from being modified. Please check that with your data privacy regulations and if possible, just copy the content here.

from californium.

Not sure, if it's a "copy & paste" issue.
Is the "-----BEGIN CERTIFICATE-----" a line on it's own? Or is the first line
"-----BEGIN CERTIFICATE----- MIIC/jCCAeagAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl ".

The Pem reader searches for line as "-----BEGIN CERTIFICATE-----", but not for lines, which contains more.
Maybe you zip the file and upload it via drag & drop?

if the format is just different to the ones I'm aware of, it's possible to adapt that.

from californium.

It's the "code" formatting issue. If I open the file with notepad++, the first line is just "-----BEGIN CERTIFICATE-----"

I removed the code formatting in the previous comment

from californium.

The bug was introduced a couple of months age, after my last tests with k8s.

Support pure certificate chains also for pem.

Bugfix is on the way.

from californium.

If possible, please retest with PR #2151

from californium.

With your changes I'm able to get a bit further. No more trust store problems and GET requests to https://kubernetes.default.svc/api/v1/namespaces/cali/pods?labelSelector=controller-revision-hash%3Dcf-extserver-a-smthsmth return 200.
I'm still having issues with the setup though - the pods are trying to discover each other. After some time there are DTLS Handshake exceptions like these:
11:03:00.984 INFO [DtlsClusterManager]: cluster-node 1 (mgmt-dtls-mac): error org.eclipse.californium.scandium.dtls.DtlsHandshakeTimeoutException: Handshake flight 1 failed! Stopped by timeout after 3 retransmissions! at org.eclipse.californium.scandium.dtls.Handshaker.handleTimeout(Handshaker.java:2070) at org.eclipse.californium.scandium.dtls.Handshaker.access$1000(Handshaker.java:128) at org.eclipse.californium.scandium.dtls.Handshaker$TimeoutPeerTask$1.run(Handshaker.java:2132) at org.eclipse.californium.elements.util.SerialExecutor$1.run(SerialExecutor.java:293) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)
And here i'm wondering maybe that's because I used a custom Californium.properties file instead of CaliforniumReceivetest3.properties that's referenced here: https://github.com/eclipse-californium/californium/blob/main/demo-apps/cf-extplugtest-server/service/Dockerfile#L22 but is not available anywhere in the project. Maybe you could add that file to the project?

from californium.

CaliforniumReceivetest3.zip

That file is generate the first time you start the server on a machine "closest to the final executing" one.
It sets up some configuration value according the number of cores and the supported cipher suites according the JCE.

For a first test, that added should also do it.

I don't guess, that the handshake exception is caused by that properties file, but we will see.
In my experience, a lot of people use really "tiny" instances with "less cpu". But java and the overall architecture of Californium requires rather bigger nodes, that works simple better ;-),

from californium.

Anyway, if it doesn't work, I will try to setup it up tomorrow to see, what currently fails.

from californium.

I see a problem with Service tyle "LoadBalancer" - service stays in "pending" state with this message
Warning SyncLoadBalancerFailed 104s (x9 over 16m) service-controller Error syncing load balancer: failed to ensure load balancer: Protocol UDP not supported by LoadBalancer

from californium.

Protocol UDP not supported by LoadBalancer

AWS EKS in 2023? What a shame ;-).

from californium.

Tried also with the provided NodePort Service type - now service status seems to be ok, but the handshake still fails.

In addition to that I'm unable to port-forward UDP ports to check if I can discover coap endpoints

from californium.

In order to use k8s with UDP, the k8s support for UDP must be given.

If AWS EKS doesn't provide that, it doesn't work.

I did the tests mainly with microk8s.
If I remember well, the single pod k8s was working with ExoScale, AWS, gcloud, Digital Ocean and Azure.
LoadBalancing for UDP maybe only on glcoud, on ExoScale with their LB, but not k8s. Digital Ocean with a 3th vm running cf-nat. AWS and Azure I don't remember.

Again, it is basically the UDP support for k8s.
The service defines, how you expose the UDP traffic to external. And the k8s internal traffic between pods must also support UDP in order to forward the UDP messages.

from californium.

FMPOV, the reported bug reading the pem truststore is fixed.

If a k8s implementation/service doesn't support UDP traffic (external/internal) then that's an issue there, Californium isn't able to fix that.

Do you have open issues left? Or could this issue be closed?

from californium.

Thanks, we can close this ticket!
Maybe just a general question - what is the current suggested approach to horizontally scale Californium server nodes using CoAP + DTLS? Either in Kubernetes or AWS cloud, or any other technology? Is it feasible to share DTLS & Connection Id state between multiple nodes using Redis or similar cache servers? Maybe there's a ready setup for this?

from californium.

what is the current suggested approach to horizontally scale Californium server nodes using CoAP + DTLS?

I still prefer the UDP forwarding based on the CID. I know others have also implemented an approach using redis to share the DTLS state (see issue #2118).
My personal focus is currently on simple deployments (single nodes) and the CoAP-S3-Proxy.

Either in Kubernetes or AWS cloud, or any other technology?

If k8s supports UDP, it's nice. If not, it's not nice. Searching the web, I'm not sure, why your experience with EKS is bad. At least what I read, UDP is supported by EKS.

It is also possible to setup cloud VMs (using cloud-init and run the service as unix systemd service, see Cf Cloud-Demo-Server ). Not sure, if the examples are easy to reproduce. Usually it requires to setup an simple http server each vm, which interacts then with the cloud-load-balancer, if that supports UDP.

Is it feasible to share DTLS & Connection Id state between multiple nodes using Redis or similar cache servers?

In my opinion, sharing the "fast changing sequence numbers" will either end up in a security concerns or poor performance. You may ask in issue #2118 to share any results.

Maybe there's a ready setup for this?

Not from my side. And if you read issue #2131 some are requesting, not to extend that advanced stuff.

from californium.

Awesome, I'll look into the mentioned resources. As for UDP in EKS, it seems that this issue is fixed, there were some missing config. Thank you for the quick fix and valuable insights!

from californium.

You're welcome.

it seems that this issue is fixed

So, does the k8s CID cluster work on EKS?

from californium.

@jmicans

Please answer the last question before closing this issue as solved.

from californium.

Thanks for the answer.

from californium.