Azure clusters in existing resource groups about gardener-extension-provider-azure HOT 19 OPEN

ok. I have updated the description of this issue and marked the vNETs as implemented.

from gardener-extension-provider-azure.

cc @dkistner @AndreasBurger

from gardener-extension-provider-azure.

Currently not. It seems, that there are still issues with orphaned iaas resources after LB service deletion exists. See here: kubernetes/kubernetes#59255
They are already working on a fix for this issue, but I'm not sure if this is the only remaining issue. Further investigation is required.

from gardener-extension-provider-azure.

FYI: https://github.com/kubernetes/community/blob/master/keps/sig-azure/0025-20180809-cross-resource-group-nodes.md

from gardener-extension-provider-azure.

@dkistner any updates here for Kubernetes 1.{9,10,11,12}?

from gardener-extension-provider-azure.

kubernetes/kubernetes#67604 Maybe relevant?

from gardener-extension-provider-azure.

@dkistner any update here?

from gardener-extension-provider-azure.

Ping... @dkistner.

from gardener-extension-provider-azure.

I think there is no update as we still see also with latest k8s versions (v1.13, 1.14) left overs in Azure, we would not get rid of them when we delete the cluster. Currently we manage the ResourceGroup and drop it when the Shoot gets deleted. All left overs will be deleted during this process. This would not be the case for existing ResourceGroups because then we would not delete them.

My advise: Wait until we see no left overs anymore and when we have enough trust that the clean up works reliable.

from gardener-extension-provider-azure.

@dkistner what I understand from the Kubernetes 1.15 release notes this scenario should become possible, right?

from gardener-extension-provider-azure.

We are currently testing if the infrastructure left overs are gone with the changes in k8s v1.15. I would recommend to re-enable this only if we are sure that we are really not experience any leaked resources. Let's wait for our test results.

from gardener-extension-provider-azure.

We will enable Azure Shoot deployments into existing vNets, which are located in a different resource group as the Shoot resources.

Current status:

• MCM adaption: gardener/machine-controller-manager#344 (done)
• Gardener extension adaption: gardener-attic/gardener-extensions#371 (done)
• Gardener: gardener/gardener#1558 (done)

In general I see with this change no reason why we should re-enable deployments in existing resource groups. We had that only because the Azure Kubernetes provider supported only deployment in existing vNets which are in the same resource group as the cluster resources and we disabled it due to the orphan resource issues (which partly still exists). Meanwhile the Azure Kubernetes provider support also deployments in existing vNets which are in other resource groups (this scenario is now implemented).
Therefore I would suggest to remove the inactive validation logic in the Gardener for the existing resource group case and close this issue afterwards. WDYT? @rfranzke, @AndreasBurger, @vlerenc

from gardener-extension-provider-azure.

@dkistner can we close this issue ?

from gardener-extension-provider-azure.

No only the deployment into existing vnets is implemented. The second part deployment into existing resource groups is still in discussion.

from gardener-extension-provider-azure.

We would be interested in this as we are currently in discussions with a customer about deploying a shoot into their Azure subscription. Being able to create resource groups seems to require quite broad permissions which we would like to avoid.

from gardener-extension-provider-azure.

As far a I recall correctly the only argument against Shoot deployment into existing resource groups are the left overs which could remain on the infrastructure when the cluster is deleted. Unfortunately the Azure cloud provider have still issues with that.
So yes, I think we can discuss to enable that.
Btw: PRs for that are warmly welcome :)

from gardener-extension-provider-azure.

So you would say it could be enabled again and there would be a disclaimer somewhere that it can result in orphaned resources?

from gardener-extension-provider-azure.

Yes, something like this I could image. Users have to be aware that some left overs can remain if the cluster is gone.

from gardener-extension-provider-azure.

@muenchdo wanna give it a try? If you need help just ping, happy to guide you

from gardener-extension-provider-azure.