Completely remove all resources for any type (listener, route, secret etc) about envoy HOT 6 OPEN

CC: @adisuissa @htuch

from envoy.

Not sure I fully understand your problem, so I'll try to answer your question, and feel free to provide more details.

From Documentation of DeltaDiscoveryResponse - It says "Removed resources for missing resources can be ignored" What does it mean exactly?

Envoy uses the xDS-protocol to subscribe to resources. There are essentially 2 types of subscriptions: collections/wildcards (CDS, LDS in Envoy work like that) and singletons (non-collections, such as EDS, SDS, RDS). Specifically in Envoy, the collections' resources can exist without having anything
In the context of a secret provided by SDS, it needs some other resource that "points to it", for example, a cluster. If that cluster is removed (say by a CDS update), and no other resource needs that secret, the cluster and the secret are removed.
Envoy will send an unsubscribe from the secret request, but the response is moot, as the server cannot override the request to unsubscribe.

from envoy.

Might be worth mentioning 2 recently related PRs: #32961 and #33362.

from envoy.

@adisuissa If I understood you then In context of SDS - If I just remove a secret but that secret is being pointed by other resource like LDS/CDS then in this case secret data will not be removed from envoy configurations. Is it correct ?

To give more details of the issue that I am facing -

I am trying to use gateway project and this installation guide - https://gateway.envoyproxy.io/latest/tasks/quickstart/

It just create resources like HTTPRoute , gateway and gatewayclass. These things gets converted to envoy proxy configurations.
To apply these resources to k8s cluster - kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/latest/quickstart.yaml -n default
Now, to remove all these configurations from k8s cluster and thus from envoy proxy, I am doing -

kubectl delete -f https://github.com/envoyproxy/gateway/releases/download/latest/quickstart.yaml -n default

But to my surprise, envoy config does not get updated instead and it keeps showing the older config. I was debugging the go-control-plane server.go. and it seems it is correctly detecting which resources are to be deleted but envoy side there is effect.

few logs from go-control-plane, these are my logs -

 send getResources []
 send getRemovedResources [httproute/default/backend/rule/0]

from envoy.

@adisuissa If I understood you then In context of SDS - If I just remove a secret but that secret is being pointed by other resource like LDS/CDS then in this case secret data will not be removed from envoy configurations. Is it correct ?

The other way around, say cluster_0 was removed and that cluster had a secret, secret_0. When cluster_0 is removed, Envoy should automatically remove secret_0. Envoy will also send a discovery request to the xDS server stating that it wants to unsubscribe from the resource. The xDS server will probably send a response that removes secret_0, but it is essentially moot, as the resource was already removed.

from envoy.

@adisuissa Thanks for the response on secret/SDS update. It is indeed helpful.

Could you please also clarify a bit on the below behavior, this does not involve any secrets. and Since I am removing everything from cluster, which includes CDS, LDS, should this not remove all related configs like RDS ?

I am trying to use gateway project and this installation guide - https://gateway.envoyproxy.io/latest/tasks/quickstart/

It just create resources like HTTPRoute , gateway and gatewayclass. These things gets converted to envoy proxy configurations. To apply these resources to k8s cluster - kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/latest/quickstart.yaml -n default Now, to remove all these configurations from k8s cluster and thus from envoy proxy, I am doing -

kubectl delete -f https://github.com/envoyproxy/gateway/releases/download/latest/quickstart.yaml -n default

But to my surprise, envoy config does not get updated instead and it keeps showing the older config. I was debugging the go-control-plane server.go. and it seems it is correctly detecting which resources are to be deleted but envoy side there is effect.

few logs from go-control-plane, these are my logs -

 send getResources []
 send getRemovedResources [httproute/default/backend/rule/0]

from envoy.