Ensure zTunnel DaemonSet is running/ready on a Node before any other pods are scheduled. about istio HOT 32 OPEN

I am not sure why we would bother with the taint if we have the init container anyways

from istio.

Why would these not work on ztunnel restart?

IMO: Denying all traffic is ok (but obviously should be avoided where possible). Traffic bypassing ztunnel entirely is not

So not super worried about ztunnel restart here (assuming we don't fail open).

CNI restart I am concerned about; we can do it right, but we need to be careful

from istio.

Inject a initcontainer need image download, could slow down application start

yes, the plan is to initially provide 2 alternatives:

• init container (as it works everywhere)
• taints (works without a webhook, but requires more setup from the user)

And long term, see if we can solve this more broader with the broader k8s community

from istio.

another option is to use what istio does today:
init containers.

Have a mutating webhook, that checks that if a pod is scheduled on a node where istio's cni is not ready, adds an init container to it

from istio.

from istio.

true! good point; we can also do both taints and init containers and let the user choose which one to use

from istio.

Good point @yuval-k and @howardjohn!

So if we do document what to taint and also add the init container to validate the istio-cni is ready on the node (similar as in sidecar's init container to validate iptables is properly configured), would this solve the concerns for beta?

cc @louiscryan @keithmattix @therealmitchconnors @hzxuzhonghu @irisdingbj

from istio.

Only for folks who prefer taint over init containers?

from istio.

Ok so two distinct modes, then.

Just to clarify, taint is not just documentation. There is an Istio-side implementation required, and there are complexities I cannot discuss publicly (for a few days only) about it (happy to discuss privately for now, I think I mentioned it on a recent call that most people in the thread were on)

from istio.

Ok I vaguely recall now, thanks! Perhaps for the immediate beta, we should just do init containers? We can add taint if user requests it later on :)

from istio.

Inject init container to each workload pod? I donot think istio should do such a hack way.
Both the init container and taint nodes do not work for the case ztunnel restart or upgrade. It will have a break in the duration.

For new k8s node scaling, I think taint may be more suitable.

from istio.

@hzxuzhonghu that's what istio does today with sidecars, so while not ideal it does have precedent

Why would these not work on ztunnel restart?
I believe the part that we need to solve for is for CNI to inject the iptables rules. If ztunnel restarts the rules are still there, so traffic will essentially be black holed.

We can start with init containers and then follow with taints, that way each user can pick what best suits them.
WDYT?

from istio.

I think we are saying the same thing.
I can take a look at adapting the init container approach to ambient.

from istio.

I think we will need a different approach to validate iptables; the current validator code seem to assume that envoy is not running (as is the case in sidecar). I'm thinking of alternative approaches and will post them here.

from istio.

my current thinking is to assume that inpod mod will merge and become the default redirection.
If that happens, we the init container can test for the existence of the ztunnel listening sockets, which is pretty easy to do.

without inpod mode this is harder, as we have no way of knowing form inside the pod that the cni is active.

from istio.

i have a PoC that seems to work here https://github.com/istio/istio/compare/master...yuval-k:istio:ambient-init-container-poc?expand=1

Once in-pod merges will I continue this and add tests.

I need a way to signal to the webhook controller that this pod is an ambient pod. Currently i'm thinking to do this using by an additional env var. i.e. The injection path will be /inject/redirection/ambient. LMK if this sounds reasonable

from istio.

Related cilium doc: https://docs.cilium.io/en/stable/installation/taints/

from istio.

i have a PoC that seems to work here https://github.com/istio/istio/compare/master...yuval-k:istio:ambient-init-container-poc?expand=1

Once in-pod merges will I continue this and add tests.

I need a way to signal to the webhook controller that this pod is an ambient pod. Currently i'm thinking to do this using by an additional env var. i.e. The injection path will be /inject/redirection/ambient. LMK if this sounds reasonable

Thanks @yuval-k - the injection path make sense. @howardjohn any comments?

from istio.

Is there a chance that node agent can validate the rule and then delay the application start

from istio.

Inject a initcontainer need image download, could slow down application start

from istio.

Is there a chance that node agent can validate the rule and then delay the application start

The race is that the node agent hasn't even started yet

from istio.

A really really bad idea that might be possible... in the CNI flow, at some point the pod IP is written to the pod spec. We could add a hook on pod UPDATE that checks if its this state, and delays it until ready.

This seems extremely unsafe though, and likely not portable

from istio.

in the CNI flow, at some point the pod IP is written to the pod spec

I believe the ip is written by kubelet, after the CNI flow is over.

What kind of hook do you mean? a k8s webhook for pod updates?
not sure how can you delay pod start from a k8s webhook?

from istio.

p.o.c branch here https://github.com/yuval-k/istio/tree/ambient-init-container-poc

from istio.

How can we ensure uninterrupted traffic for the business pods on this node when the ztunnel Pod undergoes a restart or update?

from istio.

How can we ensure uninterrupted traffic for the business pods on this node when the ztunnel Pod undergoes a restart or update?

This question is out of scope of this issue - this issue covers a race that happens when a new node is added to the cluster.
follow #48387 for your question

from istio.

How can we ensure uninterrupted traffic for the business pods on this node when the ztunnel Pod undergoes a restart or update?

This question is out of scope of this issue - this issue covers a race that happens when a new node is added to the cluster. follow #48387 for your question

@yuval-k thanks ~

I'm wondering if we really need ztunnel.

from istio.

Is it possible to dynamically update iptables scripts？ iptables rules can be executed via pilot-agent, not by istio-cni.

and ztunnel is the responsible call for triggering the iptables creation and deletion.

from istio.

There is no pilot-agent running anywhere in ambient

from istio.

There is no pilot-agent running anywhere in ambient

Yes , #49092 Just like this PR, we may be able to have something like pilot-agent.

from istio.

note that this PR adds an optional feature to istio, as some users will not want a webhook.
Why do you need to change the iptables rules?

from istio.

note that this PR adds an optional feature to istio, as some users will not want a webhook. Why do you need to change the iptables rules?

(Dynamically) configure traffic that needs to be intercepted by the mesh,instead of all traffic.

Support some annotations such as:

includeIPCidr, excludeIPCidr
includeInboudPorts, excludeInboundPorts
includeOutboutPorts, excludeOutboundPorts

from istio.